acceptodds
Under review as a conference paper at ICLR 2027

AgentVIF: Event-Aware Multi-Agent Adaptation to Dynamic Degradations in Visible–Infrared Video Fusion

Abstract

Visible-infrared video fusion aims to integrate complementary texture and thermal information into temporally consistent fused videos. However, existing methods commonly rely on fixed fusion parameters and struggle with dynamic degradations whose types and severities vary over time. We propose an event-aware multi-agent framework that couples temporal degradation perception with parameter-space expert composition. A Sentinel Agent localizes degradation transitions, while a customized Expert Agent predicts continuous multi-label degradation severities from dual-modal bursts. These predictions guide a Fusion Mixer to adaptively merge degradation-specific task vectors and generate segment-specific fusion parameters. A Feedback Agent further detects local fusion failures and triggers re-diagnosis and re-fusion. We additionally construct a 20K dual-modal burst-level instruction dataset to customize the Expert Agent. Extensive experiments demonstrate improved fusion quality, temporal consistency, and downstream tracking performance across diverse video conditions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal learning, image fusion, video fusion

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_9K92OpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an agent-based framework for visible-infrared video fusion under time-varying degradations. The framework consists of four components. A Sentinel Agent localizes the points at which the degradation state changes, and an Expert Agent predicts the severity of five degradation types (low light, over-exposure, fog, blur, and noise) for the corresponding segment. A Fusion Mixer then merges the task vectors of degradation-specific VideoFusion experts according to the predicted severities to generate segment-specific fusion parameters. Finally, a Feedback Agent detects failed segments in the fused output and triggers re-diagnosis and re-fusion. The Expert Agent is obtained by fine-tuning Qwen3-VL with LoRA on a 20K burst-level dataset built from synthetic degradations. The authors report improvements in fusion quality, temporal consistency, and object tracking performance on HDO, M3SVD, VT-MOT, and their newly constructed DIVD dataset.

Critical strengths

The core idea is fresh and intuitive. The framework diagnoses the type and severity of degradation in each temporal segment and adjusts the fusion model accordingly, which is meaningful because the design directly reflects the characteristics of the degradation. In particular, using the diagnosed severities as merging weights for degradation-specific expert models allows a segment-specific fusion model to be constructed at inference time without additional training. This is a practical way to handle compound degradations, and it has the potential to be extended to other backbones and low-level vision tasks.

Critical weaknesses
  • The contributions of the proposed method appear somewhat limited, both as a framework and as an agentic system. From the framework perspective, the paper relies heavily on existing techniques such as VideoFusion, Task Arithmetic, and TIES-Merging, and its distinguishing contribution seems to lie mainly in using the VLM-predicted severity scores as merging weights. From the agent perspective, the three proposed agents appear closer to VLM-based classifiers or regressors operating under fixed prompts and rules. Since when and which module to invoke is also determined by predefined rules, it is difficult to identify any process of autonomous planning or tool selection. In addition, the design of the severity labels, which serve as the basis for the merging weights, appears somewhat arbitrary. Native degradations in the original videos are estimated with fixed threshold rules, while the severity of synthetic degradations is defined as the normalized strength of the applied operator. As a result, there seems to be limited evidence that severity values across different degradation types correspond to the actual degree of image quality degradation.

  • The experimental validation appears limited in terms of both its scope and its baselines. First, the instruction data for the Expert Agent, the training data for the degradation-specific experts, and the DIVD evaluation set are all constructed with the same degradation operators, and the performance of the Sentinel Agent and the Expert Agent is measured only on DIVD. As a result, it remains unclear whether the proposed method generalizes to real-world settings where natural degradations vary over time. Second, several key comparisons appear to be missing. Without a comparison against a single VideoFusion model fine-tuned on all five degradation datasets jointly, it is difficult to determine whether agent-based diagnosis and expert merging are actually necessary. There is also no oracle experiment that feeds the ground-truth severity scores of DIVD into the Fusion Mixer, which makes it difficult to assess how much diagnosis errors affect the final fusion performance. Finally, the paper does not state whether the compared methods were further trained on the same degraded data, so the reported improvements may reflect a combination of additional training on degraded data and the contribution of the proposed method itself.

Questions for authors
  1. Can you provide a per-degradation analysis on natural degradation data showing whether the severity scores reflect the actual characteristics of each degradation (e.g., brightness reduction for low light and saturation for over-exposure)?

  2. Can you provide the number of failures detected and re-fusions triggered by the Feedback Agent on each dataset, along with the performance change before and after re-fusion? In addition, can you provide comparisons against a single VideoFusion model fine-tuned jointly on all five degradation datasets and an oracle setting using the ground-truth severity scores?

Reviewer_ihCMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AgentVIF, an event-aware multi-agent framework for visible-infrared video fusion under temporally varying degradations. A Sentinel Agent combines lightweight visual change cues with VLM verification to detect degradation transitions. Around each detected transition, an Expert Agent analyzes a six-frame dual-modal burst and predicts independent severity scores for low-light, over-exposure, fog, blur, and noise. These scores are used by a Fusion Mixer to compose degradation-specific VideoFusion task vectors and generate segment-specific fusion parameters. A Feedback Agent then inspects the fused sequence and triggers local re-diagnosis and re-fusion when potential fusion failures are detected.

The authors construct a 20K-sample dual-modal instruction dataset for tuning the Expert Agent and introduce DIVD, a synthetic benchmark containing temporally varying degradations. The method is evaluated on four video fusion datasets, through component ablations and agent capability tests, and on a downstream object tracking task. The paper reports improvements in fusion quality, temporal behavior, and tracking metrics compared with the evaluated baselines.

Critical strengths
  1. The paper addresses a relevant problem with a coherent system design. Temporally varying and compound degradations can require different fusion behavior within the same video. The proposed pipeline connects event detection, local multimodal diagnosis, severity-aware expert composition, and feedback-based refinement in a logically consistent way. The use of independent degradation scores is more flexible than selecting a single discrete restoration tool.

  2. The empirical evaluation covers several aspects of the proposed system. The paper compares against multiple image- and video-based fusion methods on four datasets, includes ablations for event localization, Expert Agent tuning, feedback, and expert merging, and evaluates the agents separately on DIVD. The downstream tracking experiment provides an additional application-level evaluation. Appendix B.1 also describes source-video-level separation between customization and evaluation data, which makes the reported comparisons easier to interpret.

Critical weaknesses
  1. The experiments do not isolate the contribution of the agentic adaptation mechanism, and the novelty of the Fusion Mixer is insufficiently positioned. AgentVIF trains five degradation-specific VideoFusion experts on approximately 27K degraded frame pairs per degradation type, while the main comparison uses the original VideoFusion model and other existing baselines. The ablations modify or remove agents while retaining the trained expert collection, so they do not determine how much of the improvement comes from the extra degradation-specific training rather than from event detection, VLM diagnosis, or feedback. A stronger comparison would include a single VideoFusion model jointly fine-tuned on the same degradation data, as well as a simple severity predictor controlling the same expert collection. These controls would clarify whether the VLM-based multi-agent design is necessary. In addition, the pruning, sign election, and sign-consistent parameter merging in Eqs. 7–8 closely resemble the structure of TIES-Merging. AgentVIF adds severity-dependent weighting and aggregate scaling, but the paper does not discuss this relationship or separately quantify the benefit of each modification. The pruning fraction and threshold are also not clearly specified.

  2. The evaluation does not fully validate the raw severity control signal or the claimed dynamic robustness. The Fusion Mixer uses the raw severity vector for expert activation, relative expert weights, and the aggregate adaptation scale in Eqs. 6–9. However, Table 3 evaluates baseline-subtracted and positive-clipped severity changes, as defined in Eq. 16. This evaluation can remove persistent prediction offsets that still affect the actual fusion process. Therefore, the reported diagnostic improvements do not directly establish that the raw scores used by the Fusion Mixer are reliable. The dynamic benchmark also has limited coverage. DIVD uses the same degradation operators as the instruction-tuning and expert-training data. Its events are temporally non-overlapping, and different degradation types within an event share one temporal envelope. This tests controlled temporal variation within the proposed simulator, but does not directly test unseen degradation mechanisms, naturally changing conditions, overlapping events, or independently evolving compound degradations. Finally, temporal consistency is mainly illustrated through selected temporal profiles, average luminance curves, and qualitative examples. The paper does not report dataset-level flicker measurements, motion-compensated temporal errors, parameter-switch discontinuities, or variation across sequences. The feedback stage also lacks quantitative analysis of initial and revised severity predictions, correction frequency, and computational cost.

Questions for authors
  1. Matched training controls: How does AgentVIF compare with a single VideoFusion model jointly fine-tuned on exactly the same degradation-specific training data? How does it compare with a lightweight degradation predictor controlling the same expert collection and Fusion Mixer?

  2. Raw severity calibration: Can the authors report uncorrected severity errors, false expert activations on clean or naturally degraded sequences, and downstream fusion performance under predicted, binary, and oracle severity vectors?

  3. Target definition: Appendix B.2 describes burst degradation strengths using peak synthetic strengths, whereas Eq. 18 evaluates window-averaged severity. Are the Expert Agent targets intended to represent peak severity or average severity? Please clarify how this difference affects training and evaluation.

  4. Fusion Mixer novelty: How does the proposed Fusion Mixer differ from severity-weighted TIES-style merging? Please report separate ablations for magnitude pruning, sign-conflict resolution, severity weighting, and the aggregate scale in Eq. 9.

  5. Dynamic generalization: Do the gains remain on unseen degradation operators, naturally changing degradation sequences, overlapping events, or compound degradations whose components change independently?

  6. Feedback behavior: For each feedback correction, what changes in the diagnosis, segment boundary, or fusion parameters? Please provide examples showing the initial diagnosis, revised diagnosis, and resulting output quality. It would also be useful to report how often feedback improves or degrades the result.

  7. Efficiency: What are the end-to-end runtime, VLM-call count, GPU memory usage, feedback frequency, and throughput? The Sentinel and Feedback Agents perform periodic VLM verification, so these measurements are important for assessing practical deployment.

  8. Metric protocol: Please clarify how SSIM is computed and aggregated across the visible and infrared modalities. Per-sequence variance, multiple random seeds, and statistical significance would also make the comparisons more reliable.

Reviewer_LQFUOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes AgentVIF, an event-aware multi-agent framework for visible–infrared video fusion under dynamic degradations. The Sentinel Agent localizes degradation transitions, the instruction-tuned Expert Agent predicts multi-label degradation severities, and the Fusion Mixer composes the parameters of degradation-specific experts. The Feedback Agent monitors the fused results and triggers local re-diagnosis and re-fusion. The authors also construct a 20K dual-modal video-burst instruction dataset to customize the Expert Agent. Comparisons with existing methods and ablation studies show that the framework improves fusion quality.

Critical strengths

1.The paper addresses changes in degradation type and severity over time, a setting in which fixed fusion parameters may struggle to maintain good performance throughout a video. This provides a clear motivation for event-aware adaptive fusion.

2.The modules are designed with clear logic and form a coherent pipeline for adapting fusion to compound degradations. Combining lightweight visual cues with event-triggered VLM verification also reflects an effort to avoid the cost of frame-by-frame diagnosis. Comparisons across multiple datasets and ablation studies validate the method’s effectiveness.

Critical weaknesses

1.The sources of the performance gains are not fully disentangled. The method uses both an instruction-tuned diagnostic model and separately trained degradation experts. Although the ablations demonstrate the roles of individual components, the paper lacks a comparison with a unified fusion model trained on the same degradation data, making it difficult to isolate the contribution of event-driven adaptation.

2.The Expert Agent’s severity predictions still contain errors, but the paper does not analyze how these prediction errors affect final fusion quality. In addition, the paper does not report end-to-end inference time or the number of calls to each agent, making it difficult to assess the method’s additional computational cost.

Questions for authors

1.The performance gains from event-driven adaptation and degradation-specific training are not fully disentangled. A comparison with a unified fusion model trained on the same degradation data is missing.

  1. Table 3 shows that instruction tuning improves severity estimation, but the paper does not analyze how prediction errors affect final fusion quality.

  2. The paper does not report end-to-end inference time or the number of calls to each agent, leaving the additional computational cost unclear.

  3. Some figure annotations and wording need refinement. For example, the captions of Figures 6 and 8 do not explain the ablation settings corresponding to (a)–(f), and some paragraphs contain duplicated wording.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.