- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the credit-assignment problem in omni-modal active perception and proposes MATCA and SIC. Its most valuable idea is to decompose each perception action into two questions: (1) Is the acquired content itself useful? (2) Is the selected temporal location more informative than matched alternatives? MATCA assigns separate credit to the action-type and timestamp fields, while SIC selectively transfers useful teacher actions into a reset student before renewed MATCA training. Overall, this is a conceptually clear and empirically well-supported, but computationally expensive, training framework. The method shows clear effectiveness; however, its contribution is best understood as constructing better training signals for active perception, rather than solving perception or causal attribution in a general sense.
Critical strengths
- The paper targets an important credit-assignment problem. Active perception differs from ordinary token generation because an acquisition action changes the evidence available to all subsequent decisions. The examples of useful acquisitions inside failed trajectories and harmful acquisitions inside successful trajectories clearly demonstrate why broadcasting a trajectory-level advantage to every acquisition can be misleading.
- The intervention point is carefully selected. Scoring immediately after the media return and before the next policy response helps separate the contribution of returned media from the policy's subsequent textual interpretation. The structure-matched null preserves pathway, temporal extent, media shape, and action metadata, making the intended estimand clearer than a generic state-difference reward.
- The field-level routing is technically thoughtful. Separating content and temporal-specificity credit and routing them to different structured action fields is more precise than assigning a turn-level reward to the entire response. The distinction between zero local credit and abstention is also meaningful.
- The paper combines local and global supervision without changing deployment. Reliable acquisition fields receive MATCA credit, while observations, reasoning, confidence, final answers, and abstained fields continue to receive outcome supervision. Counterfactual branches and the frozen scorer are training-only, so deployment requires no auxiliary scorer or additional tool calls.
- The evaluation includes several useful controlled comparisons. The nested MATCA ablation, 180-action continuation audit, SIC comparisons against ungated and random-gated distillation, compute-matched continued-MATCA baseline, and long-video cost analysis provide substantially more evidence than an end-to-end leaderboard table alone.
- SIC uses student-completed branches instead of unconditional imitation. Testing teacher actions under the reset student's own prefix and continuation distribution is a sensible way to filter teacher actions that may not be useful to the student. Restricting distillation to executable action-value tokens also avoids forcing the student to imitate the teacher's reasoning style.
- The supplementary material is very detailed. The paper documents eligibility checks, null construction, calibration thresholds, token-span routing, deterministic action selection, replay reconstruction, branch validity, and compute cost.
Critical weaknesses
- The estimated content effect does not fully match the action field receiving the credit. The real-versus-null contrast estimates whether content returned through the executed pathway is useful. It does not establish that the selected pathway is preferable to alternative pathways. Nevertheless, this credit is assigned to the type value. A clip may be useful relative to a masked clip while frames or audio from the same interval would be more accurate or efficient. The appendix includes a small cross-pathway diagnostic, but the training objective itself does not optimize a cross-pathway advantage.
- Content and temporal contributions are not fully disentangled. Content gain is evaluated at the executed temporal interval and therefore already depends on temporal placement. Temporal credit is then defined relative to the median of only K=3 sampled placements. This is a useful hierarchical correction, but not an orthogonal decomposition of pathway choice and temporal choice. The temporal component also lacks a direct terminal-behavior audit comparable to the content-credit audit.
- The need for SIC reveals an unresolved optimization and exploration issue, but the mechanism is not isolated. The paper argues that MATCA can reinforce only acquisitions sampled by the current policy and uses a reset student to enable renewed exploration. However, it does not directly report policy entropy, action-type coverage, timestamp coverage, or other evidence that the reset restores exploration. The SIC improvement may instead arise from selective data filtering, action-only distillation, regularization from resetting to SFT, or removal of Phase-I policy drift. The experiments show that the complete path is better than compute-matched continued MATCA, but do not establish why.
- Hard local-versus-outcome routing is not compared with plausible alternatives. MATCA already uses a hard hybrid: reliable action fields receive local credit, while other tokens and abstained fields receive outcome supervision. A soft confidence-weighted mixture, entropy or novelty bonuses, a mixture behavior policy, or same-state multi-pathway sampling could potentially address the on-policy support limitation without a separate reset-and-distill phase. Simply adding outcome and local credit may cancel the desired local sign reversal, but this does not rule out calibrated soft gating or a separate exploration bonus.
- The frozen answer probe relies on a strong calibration assumption. MATCA assumes that changes in candidate-normalized gold-answer support provide a directionally meaningful proxy for downstream task value. The 180-action audit is encouraging, but 130 of 180 pairs do not change terminal correctness, and the reported 80% agreement is conditional on the 40 gated pairs with an observable terminal flip. The audit also uses one fixed deterministic continuation policy rather than estimating expected return under the stochastic training policy.
- The semantic-null intervention may not be neutral. Black frames and silence are structure matched, but may still be out-of-distribution inputs or implicitly signal missing or failed evidence. The contrast may therefore capture both semantic evidence and the model's response to an unusual null. Null calibration mitigates this concern but does not demonstrate invariance to the null construction.
- The SIC gate may conflate action quality with continuation variance. Each teacher and student action is evaluated with a single independently sampled student continuation. Under binary terminal rewards, some accepted actions may reflect rollout noise rather than a genuine difference in expected action value. Repeated continuations, common random numbers, or a gate-repeatability analysis would strengthen this component.
- The benchmark results lack uncertainty estimates. The main tables do not report multiple independent training seeds, standard deviations, or confidence intervals. This is particularly important because the pipeline contains stochastic RL, sampled controls, hard thresholding, and a single-sample SIC gate.
- Training cost and reproducibility remain concerns. The two MATCA phases require approximately 7,200 H100-hours and SIC approximately 480 additional H100-hours, excluding Agentic SFT data generation. It is unclear whether the method remains effective with smaller scoring budgets, fewer controls, or less hardware.
- The method currently depends on multiple-choice gold labels. Although no human turn-level annotations are required, the frozen readout still requires a known correct candidate. Extending the method to open-ended tasks or settings with only learned verifiers may be nontrivial.
Questions for authors
- The content contrast measures the usefulness of the executed pathway relative to its semantic null rather than its advantage over alternative pathways. Why is this signal assigned specifically to the type field? Have you tried explicit same-state, same-window comparisons among frames, audio, and clip actions?
- Since content gain is computed at the selected temporal interval, in what precise sense are content and temporal-specificity credit factorized? Can the authors provide a paired continuation audit for temporal-specificity credit analogous to the 180-action content-credit audit?
- The paper motivates SIC as enabling renewed exploration. Can the authors provide direct evidence such as policy entropy, action-type diversity, timestamp coverage, or visitation statistics, and disentangle the effects of resetting, selective gating, and action-only distillation through appropriate controls?
- Each SIC action pair is evaluated using a single independently sampled student continuation. How stable is the acceptance decision under repeated continuations, and would multiple rollouts or common random numbers materially change the accepted set and downstream performance?
- How robust is the frozen readout and real-versus-null credit to alternative semantic-null constructions such as blurred media, shuffled frames, temporally mismatched media, or matched unrelated media?
- Why are the improvements over OmniAgent much smaller on Video-MME and VSI-Bench than on the joint audio-visual benchmarks? Category-level or capability-level breakdowns and statistical significance would help determine whether this reflects weaker dependence on active audio-visual acquisition, backbone saturation, training-distribution mismatch, or limitations of the local-credit mechanism.
7 comments
Response to pjpa (2/6): Temporal credit and window-value audit
Q3: Since content gain is computed at the selected interval, in what sense are content and temporal-specificity credit factorized? Can a paired continuation audit be provided for temporal credit, analogous to the content-credit audit?
A3: MATCA uses a hierarchical credit design. Content credit evaluates the executed return; after the content gate passes, temporal credit evaluates its placement relative to matched alternatives. The paired continuation audit covers every action with nonzero temporal credit in the 180-action sample and reproduces the content-audit pattern. Mean return differences are +0.143 for positive and −0.161 for negative temporal credit, with 75.9% sign agreement on observable flips.
Factorization. Content credit asks whether the content acquired at the executed placement helps relative to a structure-matched semantic-null return (Eq. 4). Conditional on reliable content, temporal credit asks whether the same content type at this placement performs better than at a typical matched placement: (Eq. 6). The content gate (Eq. 9) enforces this ordering and prevents harmful content from earning positive timestamp credit. Among the 127 content-gated audit actions, 42 receive positive, 29 zero, and 56 negative temporal credit. None of the 57 negative-content actions receives positive temporal credit. This hierarchy separates the two supervision questions while retaining their dependence on the executed acquisition.
New temporal audit. Following your request for an audit analogous to the content audit, we replay each of the 98 actions with nonzero temporal credit in two branches. One uses the executed interval; the other uses the median-gain matched control interval that defined . The branches share the pre-action history, pathway, media extent, and remaining budget. Both return real media, so no null enters this audit. They continue under the same frozen deterministic policy, giving :
| Temporal gate | +1 | 0 | −1 | Mean | |
|---|---|---|---|---|---|
| Positive | 42 | 9 | 30 | 3 | +0.143 |
| Negative | 56 | 4 | 39 | 13 | −0.161 |
| All | 98 | 13 | 69 | 16 | −0.031 |
Confirmed versus contradictory flips are 21.4% vs. 7.1% in the positive group and 23.2% vs. 7.1% in the negative group, closely matching the content audit (25.7% / 5.7% and 24.6% / 7.0%). Among the 29 gated actions with an observable flip, 22 agree with the credit sign (75.9%, Wilson 95% CI [57.9%, 87.8%]; two-sided exact sign test ), compared with 80.0% for content credit. The temporal sign therefore predicts whether the executed interval performs better than a matched alternative, which is the property used to supervise timestamp fields. The training ablation shows that adding temporal credit on top of content credit improves all four benchmarks (+2.1 / +0.3 / +2.4 / +1.3; Table 2). The full MATCA objective also raises LVOmniBench temporal-question accuracy by +5.7 points (Table 4). The paired temporal results provide downstream validation of this timestamp-supervision signal.
Response to pjpa (3/6): The null reference and downstream validation
Q4: How robust are the frozen readout and the real-vs-null credit to alternative null constructions, such as blurred, shuffled, temporally mismatched, or unrelated media?
A4: The structure-matched null gives MATCA a reproducible, content-free reference, with thresholds calibrated to masked-return variation in each pathway. Both deterministic and stochastic continuation audits support the downstream relevance of the resulting credit direction.
The null replaces only the semantic content of the executed return, using black frames of the same resolution and count and silence of the same duration and layout. The pathway, media shape, timestamps, and action metadata remain fixed (Appendix A.2). This removes the acquired semantic evidence while preserving the return structure, giving real-versus-null comparisons a consistent reference across actions.
We calibrate thresholds against masked-return variation for each pathway. Using 128 held-out records, we construct the content-null distribution from differences between masked executed and masked control placements. Positive and negative gates use the 95th percentile of positive values and the 99th percentile of negative magnitudes, respectively (Appendix B.1):
| Pathway | Null samples | ||
|---|---|---|---|
| Frames | 300 | 0.425 | 0.691 |
| Audio | 156 | 0.613 | 1.875 |
| Clip | 312 | 0.534 | 0.610 |
These calibrated bands determine when a signed content contrast receives credit. Audio uses the largest positive and negative thresholds, adapting the gate to its pathway-specific masked-return distribution.
The deterministic continuation audit supports the resulting credit direction, with 80.0% agreement among gated pairs with terminal-correctness flips (Table 5). The stochastic audit below tests whether this relationship persists under training-time sampling. Sensitivity to blurred, shuffled, or unrelated-media controls has not been evaluated; such comparisons would test how credit changes when the reference retains or introduces other evidence.
Q5: How should the conditional sign agreement be interpreted given the many unchanged deterministic outcomes, and does the result persist under stochastic continuations?
A5: Under the training-time stochastic policy, the positive, abstaining, and negative credit groups retain the expected ordering of mean estimated return differences. Among gated actions with nonzero estimates, 82/102 (80.4%) agree with the credit sign.
Following your question about the single deterministic continuation, we extended the audit to the training-time stochastic policy. Repeated continuations estimate each action's effect on success probability, providing a finer measure than one binary outcome. We replayed the same 180 real-versus-null pairs (T=1.0, m=8 independent continuations per branch; 2,880 rollouts) and estimated per action:
| Gate | Real succ. | Null succ. | Mean | >0 / =0 / <0 | |
|---|---|---|---|---|---|
| Positive | 70 | 64.3% | 42.9% | +0.214 | 48 / 12 / 10 |
| Abstain | 53 | 49.1% | 48.3% | +0.007 | 20 / 15 / 18 |
| Negative | 57 | 39.5% | 53.5% | −0.140 | 10 / 13 / 34 |
The positive, abstaining, and negative groups retain the ordering observed in the deterministic audit (+0.200 / 0.000 / −0.175; Table 5). Under stochastic continuations, sign agreement is 82/102 (80.4%), with exact sign-test and for the positive and negative groups. This separation between groups supports the direction used to reinforce or suppress an acquisition.
Response to pjpa (1/6): Content credit and field-level routing
We thank the reviewer for the careful assessment and for highlighting the contribution of acquisition-specific training signals. We address MATCA's credit design first, then SIC, then cost and supervision interfaces.
Q1: The real-vs-null credit scores the executed pathway, not its advantage over alternatives. Why route it to the
typefield, and were same-state, same-window pathway comparisons tried?A1: The null contrast asks whether to reinforce the executed
typedecision, which is the question addressed by an on-policy update. The same-window audit in Appendix F.1 shows that these credits also rank alternative pathways in line with terminal utility. SIC explicitly optimizes preferences between actions.A group-relative advantage determines whether to reinforce or suppress a sampled trajectory without enumerating alternatives. At the acquisition boundary, the corresponding question is whether to reinforce the executed acquisition. The real-vs-null contrast evaluates the evidence obtained through that pathway in the current context. Routing its signed credit to the
typevalue then reinforces or suppresses the acquisition choice that produced the evidence. Table 2 tests the benefit of adding this content-credit mechanism to outcome-only training.Appendix F.1 reports the requested comparison. We replay 60 frozen states through visual, acoustic, and joint branches over the identical window, scoring each branch with its own real-vs-null credit. The sign of the credit difference agrees with the terminal contrast in 48/72 pairwise comparisons (66.7%; state-clustered bootstrap 95% CI ≈ [54%, 77%]). Ranking pathways by credit also yields lower terminal regret than the executed choice (0.183 vs. 0.283).
SIC explicitly optimizes cross-action preferences during consolidation. It holds the state and student prefix fixed and compares the teacher's acquisition with the student's, often with different pathways or windows. The action whose student-completed branch earns the higher return is then transferred (§3.5, Eq. 12).
Q2: What motivates hard local-versus-outcome routing, and how does it compare with calibrated soft mixtures or exploration bonuses?
A2: Hard routing preserves the acquisition-specific update direction on reliable action fields, even when it opposes the trajectory outcome. Every other token keeps outcome supervision.
The rule has three outcomes per field (Eqs. 7–9). Content credit is issued when the real-vs-null gain clears the pathway's calibrated null band; otherwise, the field abstains and stays on outcome supervision. Temporal credit is issued only for content-reliable turns. It is zero when the placement gain lies inside its own null band and is capped at zero when content credit is negative, so a misleading acquisition cannot earn positive timestamp credit.
Local and outcome advantages are never summed on the same token because the trajectory sign can weaken or reverse the local direction. This happens in the cases the method is designed to address: a useful acquisition in a failed trajectory or a misleading one in a successful trajectory. Separate normalization of the two masks keeps the sparse action-value tokens from being diluted by the much larger outcome-token set.
The matched ablation supports this design. Content credit improves on outcome-only GRPO by +5.0 / +5.3 / +2.3 on the three joint audio-visual benchmarks, and temporal credit adds +2.1 / +0.3 / +2.4 / +1.3 on all four (Table 2). This routing also enables the cross-outcome allocation in Table 6: 36 positive credits inside failed trajectories and 29 negative credits inside successful ones. The policy-level audit on all 1,197 Daily-Omni questions confirms the intended effect on learned behavior. Compared with outcome-only GRPO under the same SFT initialization and RL budget, MATCA reduces the negative-credit share inside successful trajectories from 10.97% to 3.76% and increases the positive-credit share inside failed trajectories from 23.60% to 29.56% (new Appendix F.3).
Calibrated soft mixtures determine how credits are combined, whereas exploration bonuses and mixture samplers determine which actions are sampled. These alternatives have not been compared in training; the matched ablations establish the benefit of the implemented credit-and-routing design.