- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
EgoSPECT introduces an egocentric audiovisual benchmark organized around a perception–priority–decision cascade. The authors collect approximately 60 hours across 74 sessions and six scenarios using a low-cost headset with four raw microphone channels and a first-person camera. They evaluate three MLLMs with modality ablations, negative controls, conditional evaluation, gold event premises, and direction-of-arrival (DOA) interventions. The principal finding is that event recognition and spatial grounding are substantially less reliable than the benchmark’s premise-conditioned priority and action questions.
Critical strengths
1.The benchmark examines consequential failures beyond positive event recognition. The weak subway chime and no-approaching-vehicle controls reveal missed events and frequent false alarms. The human comparison helps establish that these tasks contain usable information. These diagnostics make the benchmark potentially useful for evaluating wearable assistants.
2.The authors investigate several alternative explanations for model performance. Question-only tests, a blind-majority baseline, gold-premise evaluation, option shuffling, and DOA interventions provide more insight than aggregate accuracy alone. The acknowledgment of event imbalance and the restriction of the gold-premise conclusion to current closed-form tasks are particularly helpful.
Critical weaknesses
1.The cascade’s downstream coverage is too narrow to strongly establish the benchmark’s central contribution. All 48 complex clips contain a smoke alarm, paired with either knocking or a doorbell. Consequently, prioritization can largely follow a fixed rule, and the disclosed action mappings are simple. A blind event-pair strategy already achieves 68.8% exact match and 84.4% partial credit. Although the paper acknowledges this limitation, it materially restricts what near-perfect downstream performance demonstrates. Furthermore, conditional accuracy selects clips that pass Q0, while dynamic scoring evaluates correctness relative to the model’s perceived situation. These are useful diagnostics, but their improvement over the gated score does not by itself isolate a general perception bottleneck. A stronger demonstration would use varied event combinations and matched interventions on the same downstream items.
2.The spatial evaluation does not adequately separate model limitations from the information preserved by each input interface. Section 4 states that Qwen’s pipeline downmixes the recording to mono, removing the inter-channel timing and level differences available to a spatial estimator. Equivalent details and validation are needed for the other models. Near-chance localization therefore cannot uniformly establish failure to reason over preserved spatial cues. The DOA-hint experiment shows that models can use explicit direction information, but it tests indoor doorbell localization rather than the emphasized problem of binding different concurrent events to their respective directions. The manuscript also omits sufficient detail about the DOA algorithm, calibration, direction annotation, and head-motion handling to assess this result confidently.
Questions for authors
1.What spatial information reaches each model after preprocessing? Please specify channel handling and provide a controlled comparison that distinguishes loss of spatial cues at the interface from failure to use available cues.
2.Can you report a matched cascade evaluation on identical clips and downstream questions, replacing predicted upstream premises with gold premises while keeping the denominator fixed? How do the conclusions change after accounting for the smoke-alarm and majority-pair priors?
3.How were event directions annotated and validated, particularly during head movement? Please specify the DOA method, array calibration, and the provenance of the 132 doorbell clips in Table 3, including their relationship to the core dataset.
4.Please reconcile the traffic decision percentages and provide the referenced per-direction results, exact human sample counts per task, and uncertainty estimates that account for repeated questions from the same clips or sessions.
8 comments
Response to Q2: Matched cascade evaluation (predicted vs. gold premise) and the blind-rule prior baseline (complete results)
We sincerely thank the reviewer for this key question. The supplementary no-gating evaluation is now complete for all three models (Q0 is answered and scored as usual, but a wrong Q0 no longer skips later questions; every event runs through its full chain; 20 sessions per model, 132 events = 84 simple + 48 complex). The earlier access difficulties caused by Google's phase-out of Gemini-2.5-Pro have been resolved. The two comparisons follow.
(1) Matched cascade evaluation: predicted vs. gold premise, same clips, same denominator
On exactly the same 48 complex clips, the downstream closed-form items are fixed at 2 per clip (CQ2 priority ranking + CQ5 action decision), giving a fixed denominator of 96 items. Under the predicted premise, CQ2's options are dynamically generated from the sounds the model itself selected in CQ1, and CQ5's stem and gold answer follow the model's CQ2 choice; if CQ1 selects fewer than 2 sounds so that no branch can be generated, the clip's 2 items score 0 with the denominator kept. Under the gold premise, ground-truth event pairs are written directly into the stem.
| Model | Predicted premise (96 items) | Gold premise (96 items) |
|---|---|---|
| Qwen2.5-Omni-7B | 93/96 = 96.9% | 96/96 = 100% |
| VideoLLaMA2.1-7B-AV | 44/96 = 45.8% | 96/96 = 100% |
| Gemini-2.5-Pro | 95/96 = 99.0% | 96/96 = 100% |
VideoLLaMA2's 45.8% decomposes completely: on the 22 clips that actually branched, its CQ2+CQ5 is 44/44 = 100%; all 52 lost items come from the 26 clips where CQ1 selected fewer than 2 sounds and no branch could be generated. Gemini's single lost item sits in a clip whose CQ1 was already partially wrong (0.5 points), so its loss likewise originates on the perception side. The entire gap between the two premises comes from the branchability of the perception output, not from ranking or decision ability.
(2) After accounting for the smoke-alarm and majority event-pair priors
We constructed the blind-rule baseline the reviewer refers to (no audio-visual input; fixed rules: Q0 answer "yes"; CQ1 answer "smoke alarm + knock"; CQ2 answer "smoke alarm"; CQ3 random direction; CQ4 answer the majority scene item; CQ5 answer "evacuate"), scored under the official rules on the same 48 clips and 288 item slots:
| Q0 | CQ1 | CQ2 | CQ3 | CQ4 | CQ5 | Chain total | |
|---|---|---|---|---|---|---|---|
| Blind-rule baseline | 100% | 84.4% | 100% | ≈12.5%–18.8% | 100% | 100% | ≈82.8%–83.9% |
| Qwen2.5-Omni-7B | 97.9% | 17.7% | 93.8% | 21.9% | 97.9% | 100% | 71.5% |
| VideoLLaMA2.1-7B-AV | 97.9% | 64.6% | 45.8% | 16.7% | 45.8% | 45.8% | 52.8% |
| Gemini-2.5-Pro | 87.5% | 88.5% | 97.9% | 17.7% | 100% | 100% | 81.9% |
(VideoLLaMA2's identical 45.8% on CQ2/CQ4/CQ5 comes from the 26 unbranchable clips scoring 0 on all three items. The baseline's CQ4 = 100% is by design — that item is a scene-understanding question whose gold answer is constant across clips and deliberately unambiguous, serving stage isolation.)
What changes and what does not:
- "Perception is the bottleneck": unchanged, and strengthened. Both open-source models score below a zero-perception blind-rule baseline — their failure is not merely "weak perception" but perception-side behavior actively losing points (Qwen's CQ1 select-all triggering the zero-score rule, 17.7% vs. the blind 84.4%; VideoLLaMA2's under-selection leaving 26 clips unbranchable). Gemini is the only model exceeding the blind baseline on CQ1 (88.5% vs. 84.4%; exact match 79.2% vs. 68.8%). No model exceeds the blind baseline in chain total. The score differences are localized to perception and its downstream branching consequences: Q0 and CQ1 vary across models, CQ3 is near-random, and VideoLLaMA2's 45.8% on CQ2/CQ4/CQ5 results from the 26 unbranchable clips rather than independent downstream failures. Thus, perception errors and their branching consequences dominate end-to-end loss.
- "Downstream is close to solved": the claim is narrowed. The reviewer's criticism holds — under the current event composition (all 48 clips contain the smoke alarm, with a fixed priority rule and action mapping), high downstream scores are achievable by a blind rule. These numbers should therefore be read as "stage decoupling works, and downstream adds no loss beyond the rule-achievable baseline when perception is correct", not as evidence of robust ranking/decision ability. We will tighten the §5.3 wording accordingly and state this scope explicitly.
- Follow-up improvement: the event-composition imbalance is already listed as a limitation in §6; the next data release will balance event pairs (reducing the smoke-alarm share) so that the blind event-pair strategy no longer works and the downstream items regain discriminative power. We will also add the blind-rule cascade baseline to the revision as a fixed reference.
We thank the reviewer again — this question prompted us to run these experiments, completing the evidence chain behind our attribution analysis.
Response to Q3: Direction annotation, verification, and the source of the 132 doorbell clips
We sincerely thank the reviewer for raising this point, and we apologize for the insufficient clarity in the manuscript. We explain honestly and provide additional data below.
1. How event directions were annotated. Directions are head-relative bearings, recorded over each event's time span with one annotation at the event's onset and one at its end. The direction question (Q2) asks from which direction the sound initially came, and the label semantics are aligned with the question's definition — we evaluate static localization at the event onset, not continuous bearing tracking.
2. An honest statement of the limitation (head motion). The reviewer's concern is valid: the wearer may turn their head within an event window, and the two onset/offset annotations cannot rule out mid-event changes. We have not systematically verified head motion during events, and this is the boundary of the current annotation scheme. The recording device carries an IMU that continuously logs head pose; these data will be released with the dataset. Should dynamic bearing tracking be needed, we will build an advanced annotation scheme with continuous head-motion calibration.
3. DOA method and array calibration. The DOA estimator is GCC-PHAT with an SRP-PHAT search (50 ms frames, 25 ms hop, six microphone pairs, 1° azimuth scan, energy VAD with modal 10° histogram aggregation). The array geometry is taken from the official XVF3800 AEC_MIC_ARRAY_GEO — four microphones at (±33 mm, ±33 mm), a 66 mm square, max inter-channel delay ≈ 4.3 samples at 16 kHz — and this official specification serves as the basis for array calibration. Full estimator parameters, channel mapping, and azimuth conventions are given in our response to Question 1.
4. The source of the 132 doorbell clips and their relationship to the core dataset. The 132 doorbell clips used in the DOA intervention (Table 7) come from an independent direction-recording dataset, collected specifically for the controlled direction intervention and separate from the six core scenarios. The eight directions are covered in a balanced way: front 16, front-right 18, right 16, rear-right 16, rear 17, rear-left 16, left 17, front-left 16 (12.1%–13.6% per direction, close to uniform). Each session contains a single source at a fixed, known azimuth (recorded per session in the metadata), so stable direction ground truth is available without event-onset alignment.
5. Verification: per-direction model behavior and a human baseline. On this balanced dataset, all three models under MLLM-alone remain close to the 12.5% random baseline overall (Qwen 14.4% AV, Gemini 7.6% AV, VideoLLaMA 9.8% AV; all 12.9% under AO). The per-direction breakdown further reveals a consistent failure pattern: each model's few correct answers concentrate on one or two fixed directions (Qwen on "front/rear," Gemini on "front-left," VideoLLaMA on "front"), with nearly all remaining directions wrong — a directional response prior rather than localization ability (the full per-direction table is in our response to Question 4). In addition, we measured a human baseline on the direction task: 10 participants, 100 items in total, 92% overall accuracy, no direction below 76.9%, with errors concentrating on rear directions — consistent with the known difficulty pattern of human auditory localization. The task and its labels are fully solvable for humans, which further corroborates the reliability of the direction labels.
We thank the reviewer again for the careful reading — the suggestion has genuinely made the evidence chain behind our direction-related conclusions more complete.
Response to Q1: Channel handling, and a controlled comparison of interface loss vs. unused cues
We sincerely thank the reviewer for this incisive question. We answer in three parts: channel handling, the controlled comparison, and the DOA details.
1. In what form does spatial information reach each model after preprocessing?
Upon verification, all three evaluated models discard inter-channel information at the input stage — they just discard it differently:
librosa.load(mono=True): channels averagedwav[:, 0]: only channel 0 keptThe specific evidence:
qwen-omni-utilstoolchain loads audio withlibrosa.load(sr=16000), whosemonodefaults toTrue, averaging all channels; passing a multi-dimensional numpy array directly raises"Support only mono audio", so the official interface does not support multi-channel input. (Code:qwen-omni-utils/src/qwen_omni_utils/v2_5/audio_process.py.)(N, C)is reduced towav[:, 0](channel-dropping, not mixing); audio from video is forced to mono via ffmpeg-ac 1. The signal is then resampled to 16 kHz and converted to 128-dim Kaldi fbank features for the BEATs encoder, at which point inter-channel time/level differences are unrecoverable. (Code:videollama2/mm_utils.pyat commit8cedfbd.)Two statements in §4 of the original manuscript are therefore incorrect ("we feed the same four-channel raw microphone audio to every model" and "Gemini receives the raw channels natively"), and the revision will correct them to the verified description above.
2. A controlled comparison: interface loss vs. unused cues.
The DOA intervention (Table 7) provides a controlled, laddered comparison that bears on the two causes raised by the reviewer:
Levels 3–4 show that models can exploit spatial cues supplied as text, identifying reliable cue acquisition as the bottleneck under the evaluated mono interfaces. However, interface stripping and model-level spatial-cue extraction remain confounded. The revision will therefore scope our spatial conclusions to "models via their actual interfaces" and list natively multi-channel models as future work (Appendix H). Native channel-preserving evaluation is needed to isolate the two causes; if useful, we can run an endpoint calibration with known-direction panned stimuli.
3. DOA method and array details.
DOA estimator. GCC-PHAT with an SRP-PHAT search: 50 ms frames, 25 ms hop, 16 kHz; per frame, PHAT-weighted generalized cross-correlation over all six microphone pairs (nfft = 16384), scanning azimuths 0°–359° in 1° steps and summing correlations at the theoretical pair delays; the argmax gives the frame-level estimate. Frames below 1.5× the segment mean energy are discarded (energy VAD on one reference channel), and the clip-level estimate is the modal 10° histogram-bin center over surviving frames (more robust to onset transients than a median).
Array geometry. Four microphones at (±33 mm, ±33 mm) relative to the array center — a square of 66 mm side (93 mm diagonal; max inter-channel delay ≈ 4.3 samples at 16 kHz) — from the official XVF3800 AEC_MIC_ARRAY_GEO. The x-axis points right, y-axis forward. Channel mapping (0-based): ch2→rear-right, ch3→rear-left, ch4→front-left, ch5→front-right. Azimuth is measured clockwise from the front (front = 0°, right = 90°, …, front-left = 315°).
Hint-generation window. The DOA hint is computed over the full 60 s clip (trailing segments < 3 s discarded). The direction recordings are controlled: each session contains a single source at a fixed, known azimuth (recorded in the session metadata), so the whole-clip mode is a stable estimate with no need for event-onset alignment.
We thank the reviewer again for the careful reading and constructive comments, which have genuinely helped us improve the accuracy of the paper's presentation.