acceptodds
Under review as a conference paper at ICLR 2027

Reasoning in the Dark: Event-RGB Adaptation for Low-Light Video MLLMs

Abstract

Video multimodal large language models (video MLLMs) rely heavily on RGB appearance cues and degrade sharply in low light, where texture, color, and object boundaries are lost. Event cameras provide complementary motion and temporal information under poor illumination, but integrating event streams into language-based reasoning is bottlenecked by scarce paired event-language supervision, especially under low light. We propose LITE-Event, a sample-efficient framework for low-light video reasoning that fuses event and RGB streams. A lightweight visual-language interface combines modality-specific projectors with gated fusion. A support-conditioned hypernetwork then generates LoRA-style residual updates from a small non-target support set, adapting the interface without updating the frozen encoders or LLM. We also introduce three resources for training and evaluation: NExT-QA-Dark, a large-scale synthetic low-light RGB-event dataset for pretraining, EventLang-Dark, a curated event-language dataset for adaptation, and RealDark-Bench, a real-world low-light RGB-event reasoning benchmark. Trained predominantly on synthetic data, LITE-Event reaches 85.0% on RealDark-Bench, outperforming the strongest enhancement-then-reason baseline by 5.2 points and surpassing both RGB-only and event-only video MLLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal large language models, event-based vision, low-light vision, video reasoning, RGB-event fusion, parameter-efficient adaptation, hypernetworks, vision-language learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RQoWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LITE-Event for low-light video understanding using synchronized RGB and event inputs. It combines frozen visual encoders and an LLM with gated fusion and a support-conditioned hypernetwork that generates low-rank interface updates. The authors construct synthetic pretraining data, a real-event adaptation dataset, and RealDark-Bench for evaluation.

Critical strengths
  1. The paper addresses a practical problem: using complementary event information when RGB observations become unreliable under low light.

  2. The evaluation includes real low-light recordings from sources disjoint from the adaptation data. The overall improvements over RGB-only, enhancement-based, and event-only baselines provide evidence that the combined system is useful.

Critical weaknesses
  1. All reported QA evaluations use benchmarks or test splits constructed by the authors. Although the target recordings come from disjoint sources, no results are reported on existing public benchmarks under their original evaluation protocols. This makes it difficult to assess how much the results depend on the particular data selection and annotation process.

  2. LITE-Event is evaluated only with CM3AE encoders and Llama-3.1-8B-Instruct. Comparisons against Qwen and InternVL do not establish that the proposed interface also works with those backbones.

  3. Generalization varies considerably across data sources. On HUE, LITE-Event achieves 65.3%, below InternVL3 at 78.7% and Qwen3-VL at 77.3%. The authors acknowledge this limitation, but the explanation based on appearance-heavy scenes is not experimentally established.

  4. The main external baselines use either RGB or events, whereas LITE-Event receives both. These comparisons therefore do not fully separate the benefit of additional sensory information from that of the proposed method.

Questions for authors
  1. Is there evidence that the gains persist with another encoder or LLM backbone?
  2. How does LITE-Event compare with a strong RGB–event fusion baseline using the same encoders, LLM, and training data on RealDark-Bench?
5 comments

We thank the reviewer for the detailed feedback and for recognizing the practical motivation of our work. In response, we conducted additional experiments addressing backbone generalization, matched RGB-event adaptation baselines, and the performance gap on HUE. These analyses provide further evidence for the proposed adaptation mechanism while clarifying its limitations across data sources. We also acknowledge the need for evaluation on independently established benchmarks and discuss the scope of our current evaluation below.

W1. Evaluation Beyond Author-Constructed Annotations

We thank the reviewer for raising this important point. We agree that evaluation on an independently established benchmark under its original annotation and evaluation protocol would provide an additional test of generalization.

To our knowledge, we are not aware of an established benchmark that simultaneously provides real low-light RGB video, temporally aligned event streams, and video-level reasoning annotations, which is the input/task setting studied here. This gap motivated the construction of RealDark-Bench.

Importantly, while the QA annotations are constructed by us, the underlying recordings are not. RealDark-Bench is built from four independently collected public RGB-event datasets, EventVOS, SDE, RELED, and HUE, comprising 600 questions across seven reasoning categories. All questions are independently verified by three annotators, with agreement from at least two required for inclusion. Moreover, all four recording sources are disjoint from EventLang-Dark, and no RealDark-Bench sample is used for training, support construction, model selection, or tuning. We also report results separately for all four constituent datasets rather than relying only on aggregate accuracy, thereby exposing performance variation across recording sources.

We agree that the author-constructed QA annotations and the absence of a fully independent benchmark with the same input/task formulation remain limitations. We will make this scope explicit in the revised manuscript and identify evaluation on independently established low-light RGB-event reasoning benchmarks, as they become available, as an important direction for future work.

W2/Q1. Generalization to a Second LLM Backbone

We thank the reviewer for highlighting the need to test whether the proposed adaptation mechanism generalizes beyond Llama-3.1-8B-Instruct. We therefore implemented LITE-Event with a second LLM, Qwen3-VL-8B-Instruct, while keeping the frozen visual encoders, RGB-event fusion interface, support-conditioned hypernetwork, and training protocol unchanged.

To isolate the contribution of the adaptation mechanism, we also evaluated a controlled static RGB-event LoRA baseline under the same Qwen backbone and experimental setup:

LLM backboneStatic RGB-event LoRALITE-EventImprovement
Llama-3.1-8B-Instruct82.6785.00+2.33 pp
Qwen3-VL-8B-Instruct69.5078.33+8.83 pp

With the Qwen3 backbone, LITE-Event achieves 78.33% (470/600) compared with 69.50% (417/600) for Static LoRA baseline, an improvement of 8.83 percentage points, or 53 additional correct answers. The improvement is observed across all seven question categories.

Thus, although absolute accuracy varies across LLMs, support-conditioned adaptation improves over static RGB-event LoRA with both tested backbones: +2.33 pp with Llama and +8.83 pp with Qwen. This provides direct evidence that the adaptation gain is not specific to a specific LLM backbone.

We will include this second-backbone evaluation in the revised manuscript. This experiment establishes generalization across LLM backbones, while generalization to alternative visual encoders remains to be evaluated.

W3. Diagnosing the HUE Generalization Gap

We thank the reviewer for this observation and agree that our original explanation of the HUE result required further evidence. We therefore analyzed all 75 HUE questions by category to determine where the performance gap arises.

The gap is strongly concentrated in object-recognition questions. On the 37 object questions, LITE-Event achieves 59.5% (22/37), compared with 81.1% (30/37) for InternVL3 and 78.4% (29/37) for Qwen3-VL. These object questions account for 8 of the 10 additional errors relative to InternVL3 (80%) and 7 of the 9 relative to Qwen3-VL (77.8%).

HUE subsetLITE-EventInternVL3Qwen3-VL
Object (37)59.581.178.4
Non-object (38)71.176.376.3
Overall (75)65.378.777.3

When object-recognition questions are excluded, the gap decreases from 13.3 to 5.3 percentage points relative to InternVL3 and from 12.0 to 5.3 points relative to Qwen3-VL. Thus, the weaker HUE result is not a uniform source-wide degradation, but is primarily localized to the object-recognition subset.

We will revise the manuscript accordingly, replacing the broader “appearance-heavy scenes” explanation with this empirically supported characterization and explicitly discussing object recognition on HUE as a limitation.

W4/Q2. Separating Multimodal Input from Support-Conditioned Adaptation

We thank the reviewer for raising this important concern. We evaluated the adaptation baselines from Table 3 on RealDark-Bench using the same RGB-event inputs, frozen visual encoders, frozen LLM, and training data as LITE-Event, thereby separating the benefit of additional sensory information from that of the proposed support-conditioned adaptation.

Model / AdaptationInputAccuracy (%)
Projector from scratchRGB + Event75.50
Fine-tuned projectorRGB + Event80.83
Static LoRA projectorRGB + Event82.67
LITE-Event (RGB-only)RGB80.17
LITE-Event (Event-only)Event82.00
Full LITE-EventRGB + Event85.00

The results disentangle the two effects. First, multimodal sensing contributes to performance: full LITE-Event improves over its RGB-only and event-only variants by 4.83 and 3.00 percentage points, respectively. Second, when both modalities and the training setup are held fixed, LITE-Event still outperforms the strongest controlled RGB-event baseline, Static LoRA, by 2.33 points (85.00% vs. 82.67%; 510 vs. 496 correct answers). This is consistent with the 2.40-point advantage previously observed on EventLang-Dark.

Thus, the gain cannot be attributed to access to both RGB and event inputs alone. Multimodal fusion provides one source of improvement, while support-conditioned adaptation provides an additional gain over a static RGB-event adapter.

Thank you to the authors for the detailed experiments, I still have a few concerns:

  1. In the added experiments, after Qwen3-VL-8B-Instruct is fine-tuned with LoRA, why is it lower than the original backbone? I think this is counter-intuitive, and I hope the authors can provide the detailed parameter configuration of LoRA and explanation.
  2. I see that the authors have not updated the paper. If the paper can be updated, it will be more convincing. If the authors can resolve these concerns, I will raise my score.
Reviewer_nzbqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a novel approach to leverage the rich spatio-temporal information from event cameras to complement degraded RGB in low-light video understanding with MLLMs. This motivates the introduction of a pipeline for generating synthetic training data and a dataset for adaptation with real event data. The paper also introduces a benchmark and a sample-efficient framework for low light RGB-event reasoning in videos.

Critical strengths

The paper introduces a novel benchmark for an open problem for low-light cross domain rgb-event understanding. LITE-Event demonstrates that event features adds relevant complementary information that helps MLLMs over RGB only or even RGB enhancement techniques in low-light settings.

Critical weaknesses

In eqn 7, averaging A and B independently may not reproduce the average of the generated updates, since mean(A).mean(B)!=mean(AB). Thus the authors should either average the product or show that in the current setting both converge.

Questions for authors
  1. Please provide the language bias in the benchmark by evaluating with only an LLM.
  2. See critical weakness.
2 comments

We thank the reviewer for the positive assessment of our work and for the insightful suggestions. We have addressed both concerns through additional experiments: a direct comparison of factor-wise and update-wise LoRA aggregation strategies, and a language-only evaluation of RealDark-Bench with answer-option shuffling. We also clarify the mathematical distinction in Eq. 7 and acknowledge the language priors identified in the benchmark. Our detailed responses follow.

Q1. Language Bias in RealDark-Bench

We thank the reviewer for this suggestion. To quantify language-only predictability in RealDark-Bench, we evaluated Llama-3.1-8B-Instruct using only the question text and answer options, with no RGB or event inputs. We additionally repeated the evaluation after independently shuffling the answer choices with three random seeds.

LLM-only settingCorrect / TotalAccuracy (%)
Original option order348/60058.00
Shuffled (seed 17)353/60058.83
Shuffled (seed 29)364/60060.67
Shuffled (seed 43)357/60059.50
Full LITE-Event510/60085.00

These results show that RealDark-Bench contains substantial language-only predictability, with text-only performance well above the 25% random-choice baseline. However, performance remains similar after answer-order shuffling, suggesting that this signal is not primarily driven by fixed option-position artifacts.

At the same time, the visual modalities provide substantial additional information: full LITE-Event reaches 85.0%, compared with 58.0% using the same LLM with text alone, a 27.0-point improvement. Thus, although RealDark-Bench is not free of language priors, text alone leaves a large performance gap that is recovered when RGB-event evidence is provided.

We will explicitly report both this limitation and the LLM-only control in the revised manuscript.

W1. Factor-wise versus Update Averaging

We thank the reviewer for pointing out this important distinction. We agree that mean(A) × mean(B) ≠ mean(AB) in general. Our implementation intentionally performs factor-wise averaging: the generated A and B factors are averaged separately and then composed to form a single rank-r residual projector. We did not intend Eq. 7 to imply mathematical equivalence to averaging the resulting weight updates.

To directly assess the empirical consequence of this design choice, we evaluated both aggregation strategies on RealDark-Bench using the same trained model and identical source support batches:

Aggregation methodCorrect / TotalAccuracy (%)
Factor averaging: mean(A) × mean(B)510/60085.00
Update averaging: mean(AB)505/60084.17

Directly averaging the effective updates does not improve performance: factor-wise averaging is slightly better, by 0.83 percentage points (5 questions). Factor-wise averaging also preserves the intended rank-r residual parameterization, whereas averaging the full A⁽ᵐ⁾B⁽ᵐ⁾ updates need not remain rank-r.

Thus, although the two aggregation rules are mathematically distinct, the alternative proposed by the reviewer does not yield an empirical advantage in our setting. We will revise Eq. 7 and the surrounding text to explicitly describe our procedure as factor-wise averaging, rather than averaging the generated updates.

I thank the authors for their rebuttal and quantifying the language bias in the benchmark. The update in the manuscript regarding equation 7 addresses my concern. Therefore, I will keep my positive score.

Reviewer_CyZdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LITE-Event, which combines low-light RGB video with event-camera signals through a small trainable interface, while keeping the visual encoders and language model frozen. It introduces a staged training approach using synthetic data and limited real-event supervision, alongside RealDark-Bench for evaluation on real low-light scenes. Experiments show higher overall question-answering accuracy than the tested RGB-only, image-enhancement, and event-only baselines.

Critical strengths
  1. It proposed RealDark-Bench, which provides human-reviewed questions on paired low-light RGB and event recordings from multiple sources. Its explicit separation from training and support data enables a meaningful test of transfer to unseen low-light scenes.

  2. Promising results with limited real-event supervision. After synthetic pretraining and adaptation on real-event sequences, LITE-Event reaches higher accuracy on RealDark-Bench, versus the strongest reported baseline. This supports the practical value of combining event and RGB information when real-event language annotations are scarce.

Critical weaknesses
  1. One important baseline is missing. On RealDark-Bench, the paper does not compare against a simpler model using the same RGB and event inputs and training data. It is therefore unclear whether the gains come from the proposed method or mainly from adding event information. The simpler adapter comparison is reported only on EventLang-Dark, where the proposed method answers four more questions correctly out of 167.

  2. The claim of needing less training data is not established. Using few real-event examples is encouraging, but the paper does not show how competing methods perform with different amounts of training data. It therefore cannot establish that its approach learns more efficiently.

  3. Better temporal reasoning is not clearly demonstrated. About 60% of the benchmark concerns objects and spatial relationships, and the method does not lead on temporal questions. Without testing single-frame inputs or shuffled frame order, the gains may reflect better visual recognition rather than better understanding of events over time.

Questions for authors

Please refer to Weaknesses

5 comments

We thank the reviewer for the constructive feedback and for recognizing the value of RealDark-Bench and our synthetic-to-real adaptation framework. We have conducted additional experiments to directly address all three major concerns: matched RGB-event adaptation baselines on RealDark-Bench, comparative sample-efficiency analysis, and temporal-input ablations. These results provide stronger evidence for the contribution of support-conditioned adaptation and clarify the role of temporal information in LITE-Event. We address each point below.

W1. Controlled RGB-Event Baselines on RealDark-Bench

We thank the reviewer for identifying this important missing control. The new controlled RGB-event comparisons confirm that LITE-Event's gain is not explained by access to both modalities alone.

We evaluated the adaptation baselines from Table 3 on RealDark-Bench using the same RGB-event inputs, frozen visual encoders, fusion interface, frozen LLM, and NExT-QA-Dark/EventLang-Dark supervision as LITE-Event, while replacing support-conditioned adaptation with simpler alternatives.

Model / AdaptationInputCorrect / 600Accuracy (%)
Projector from scratchRGB + Event45375.50
Fine-tuned projectorRGB + Event48580.83
Static LoRA projectorRGB + Event49682.67
LITE-Event (RGB-only)RGB48180.17
LITE-Event (Event-only)Event49282.00
Full LITE-EventRGB + Event51085.00

These results disentangle the two effects. Multimodal sensing is beneficial: full LITE-Event improves over its RGB-only and event-only variants by 4.83 and 3.00 percentage points, respectively. More importantly, when both sensory input and training supervision are matched, LITE-Event still outperforms the strongest simpler RGB-event baseline, Static LoRA, by 2.33 points (510 vs. 496 correct answers). Thus, the gain cannot be attributed to additional sensory information alone. RGB-event fusion contributes to performance, while support-conditioned adaptation provides an additional improvement over a matched static RGB-event adapter.

The same pattern is observed on EventLang-Dark, where LITE-Event exceeds the matched Static LoRA baseline by 2.40 points, indicating that the advantage persists from source-domain evaluation to held-out real-world cross-dataset transfer.

We will add these matched RealDark-Bench controls to the revised manuscript.

W2. Sample Efficiency under Limited Real-Event Supervision

We thank the reviewer for this important observation. To directly test sample efficiency, we compared LITE-Event with the static LoRA projector baseline from Table 3 while systematically reducing the amount of real-event supervision.

We vary only the Stage-3 training data (25%, 50%, and 100%), while keeping Stage-1/2 pretraining, frozen encoders, frozen LLM, and evaluation protocol unchanged. The subsets are sampled deterministically at the video level while preserving the source distribution.

Stage-3 Training DataLoRA-adapted ProjectorLITE-EventImprovement
25%68.3374.17+5.83 pp
50%73.6777.83+4.17 pp
100%82.6785.00+2.33 pp

LITE-Event consistently outperforms static LoRA adaptation at every supervision level, and its advantage becomes larger as real-event supervision decreases, from +2.33 pp with the full training set to +5.83 pp with only 25%. More notably, LITE-Event with 25% of the Stage-3 data (74.17%) already slightly exceeds the LoRA baseline trained with 50% (73.67%), despite using half as much real-event supervision.

These results provide direct evidence that support-conditioned adaptation is more effective than static LoRA in the low-data regime, rather than the observed gains arising simply from access to additional real-event training data. We will include this data-scaling analysis in the revised manuscript.

W3. Dependence on Temporal Evidence and Ordering

We thank the reviewer for this important observation. We agree that the original results did not isolate whether LITE-Event benefits from temporal structure or primarily from stronger visual recognition. We therefore evaluated additional single-frame and temporal-order ablations on RealDark-Bench while keeping all model weights fixed.

Each RGB frame Iₜ is paired with an event representation Eₜ formed by accumulating the events occurring between Iₜ and the next RGB frame Iₜ₊₁.

We test two complementary perturbations: (i) retaining only the first or middle RGB frame together with its associated inter-frame event representation, and (ii) shuffling the sequence of these aligned pairs. The first removes evidence across multiple frames and event intervals while retaining the local temporal information within the selected interval. The second preserves all RGB and event representations and their cross-modal correspondence while disrupting only their temporal order.

CategoryFull LITEFirstMiddleShuffled
Spatial87.862.263.367.2
Object87.465.063.472.7
Temporal73.562.766.366.3
Visibility97.188.285.376.5
Action79.269.869.871.7
Overall85.066.266.368.3

Removing multi-frame evidence reduces overall accuracy from 85.0% to 66.2–66.3%. More importantly, temporal shuffling reduces accuracy by 16.7 points, from 85.0% to 68.3%, even though the same RGB and event observations remain available and cross-modal alignment is preserved. This provides direct evidence that the model uses temporal ordering rather than relying only on static visual evidence.

On questions explicitly labeled as temporal, accuracy also decreases from 73.5% to 66.3% under shuffling. The effect is not confined to this category, however, because spatial, visibility, and action questions can also require integrating evidence across multiple frames and time steps.

We will revise the paper to make the more precise claim that LITE-Event benefits from multi-frame temporal evidence and temporal ordering, rather than claiming superiority specifically on the temporal-reasoning category.

We thank the reviewer again for the constructive feedback. We had uploaded a revised manuscript incorporating the additional experiments presented in our rebuttal, including controlled RGB-event baselines, sample-efficiency comparisons, and temporal-input interventions.

We would be grateful to know whether these new results and revisions have addressed your main concerns, or whether any specific issues remain. We hope the additional evidence helps inform your assessment and would welcome further discussion while the discussion period remains open.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.