acceptodds
Under review as a conference paper at ICLR 2027

EgoSpect: Egocentric Spatial Perception Event-Cascade Testbench

Abstract

EgoSPECT is a first-person audio-visual benchmark for evaluating event-driven embodied intelligence across perception, priority assessment, and behavioral decision-making. Unlike benchmarks that evaluate these capabilities in isolation, EgoSPECT uses explicitly grounded question chains to measure the full event cascade while localizing failures to individual stages. We collect approximately 60 hours of egocentric recordings across six indoor and outdoor scenarios and evaluate three multimodal large language models under audiovisual, audio-only, video-only, and text-only conditions. Across the evaluated tasks, perception is the primary bottleneck: models frequently miss weak or masked events, confuse event categories, and over-report ambient sounds as hazards. In the complex multi-event scenario, end-to-end accuracy of 27.1%–62.8% rises to 69.0%–79.2% under conditional evaluation, while the three tested models reach 100% on 96 closed-form downstream questions when the ground-truth event pair is supplied as a premise. Eight-way spatial direction estimation remains near chance, but controlled DOA intervention substantially improves performance, with Oracle DOA revealing further headroom. These results show that robust embodied perception requires not only recognizing events, but also distinguishing relevant events from ambient sound and jointly grounding event semantics with spatial location under noisy and concurrent conditions.

Primary area:
datasets and benchmarks
Keywords:
egocentric vision, audio-visual learning, embodied AI, multimodal dataset, head-mounted sensing, audio-visual event localization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TCpfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EgoSPECT introduces an egocentric audiovisual benchmark organized around a perception–priority–decision cascade. The authors collect approximately 60 hours across 74 sessions and six scenarios using a low-cost headset with four raw microphone channels and a first-person camera. They evaluate three MLLMs with modality ablations, negative controls, conditional evaluation, gold event premises, and direction-of-arrival (DOA) interventions. The principal finding is that event recognition and spatial grounding are substantially less reliable than the benchmark’s premise-conditioned priority and action questions.

Critical strengths

1.The benchmark examines consequential failures beyond positive event recognition. The weak subway chime and no-approaching-vehicle controls reveal missed events and frequent false alarms. The human comparison helps establish that these tasks contain usable information. These diagnostics make the benchmark potentially useful for evaluating wearable assistants.

2.The authors investigate several alternative explanations for model performance. Question-only tests, a blind-majority baseline, gold-premise evaluation, option shuffling, and DOA interventions provide more insight than aggregate accuracy alone. The acknowledgment of event imbalance and the restriction of the gold-premise conclusion to current closed-form tasks are particularly helpful.

Critical weaknesses

1.The cascade’s downstream coverage is too narrow to strongly establish the benchmark’s central contribution. All 48 complex clips contain a smoke alarm, paired with either knocking or a doorbell. Consequently, prioritization can largely follow a fixed rule, and the disclosed action mappings are simple. A blind event-pair strategy already achieves 68.8% exact match and 84.4% partial credit. Although the paper acknowledges this limitation, it materially restricts what near-perfect downstream performance demonstrates. Furthermore, conditional accuracy selects clips that pass Q0, while dynamic scoring evaluates correctness relative to the model’s perceived situation. These are useful diagnostics, but their improvement over the gated score does not by itself isolate a general perception bottleneck. A stronger demonstration would use varied event combinations and matched interventions on the same downstream items.

2.The spatial evaluation does not adequately separate model limitations from the information preserved by each input interface. Section 4 states that Qwen’s pipeline downmixes the recording to mono, removing the inter-channel timing and level differences available to a spatial estimator. Equivalent details and validation are needed for the other models. Near-chance localization therefore cannot uniformly establish failure to reason over preserved spatial cues. The DOA-hint experiment shows that models can use explicit direction information, but it tests indoor doorbell localization rather than the emphasized problem of binding different concurrent events to their respective directions. The manuscript also omits sufficient detail about the DOA algorithm, calibration, direction annotation, and head-motion handling to assess this result confidently.

Questions for authors

1.What spatial information reaches each model after preprocessing? Please specify channel handling and provide a controlled comparison that distinguishes loss of spatial cues at the interface from failure to use available cues.

2.Can you report a matched cascade evaluation on identical clips and downstream questions, replacing predicted upstream premises with gold premises while keeping the denominator fixed? How do the conclusions change after accounting for the smoke-alarm and majority-pair priors?

3.How were event directions annotated and validated, particularly during head movement? Please specify the DOA method, array calibration, and the provenance of the 132 doorbell clips in Table 3, including their relationship to the core dataset.

4.Please reconcile the traffic decision percentages and provide the referenced per-direction results, exact human sample counts per task, and uncertainty estimates that account for repeated questions from the same clips or sessions.

8 comments
AuthorsOpenReview ↗

Response to Q1: Channel handling, and a controlled comparison of interface loss vs. unused cues

We sincerely thank the reviewer for this incisive question. We answer in three parts: channel handling, the controlled comparison, and the DOA details.

1. In what form does spatial information reach each model after preprocessing?

Upon verification, all three evaluated models discard inter-channel information at the input stage — they just discard it differently:

ModelProcessingSignal reaching the model
Qwen2.5-Omni-7Blibrosa.load(mono=True): channels averaged16 kHz mono (average of 4 mics)
Gemini-2.5-ProChannels combined on the API sideMono (per official documentation)
VideoLLaMA2.1-7B-AVwav[:, 0]: only channel 0 kept16 kHz mono (a single microphone)

The specific evidence:

  • Qwen2.5-Omni-7B. The official qwen-omni-utils toolchain loads audio with librosa.load(sr=16000), whose mono defaults to True, averaging all channels; passing a multi-dimensional numpy array directly raises "Support only mono audio", so the official interface does not support multi-channel input. (Code: qwen-omni-utils/src/qwen_omni_utils/v2_5/audio_process.py.)
  • VideoLLaMA2.1-7B-AV. A directly read wav (N, C) is reduced to wav[:, 0] (channel-dropping, not mixing); audio from video is forced to mono via ffmpeg -ac 1. The signal is then resampled to 16 kHz and converted to 128-dim Kaldi fbank features for the BEATs encoder, at which point inter-channel time/level differences are unrecoverable. (Code: videollama2/mm_utils.py at commit 8cedfbd.)
  • Gemini-2.5-Pro. Google's official Audio understanding documentation states: "If the audio source contains multiple channels, Gemini combines those channels into a single channel."

Two statements in §4 of the original manuscript are therefore incorrect ("we feed the same four-channel raw microphone audio to every model" and "Gemini receives the raw channels natively"), and the revision will correct them to the verified description above.

2. A controlled comparison: interface loss vs. unused cues.

The DOA intervention (Table 7) provides a controlled, laddered comparison that bears on the two causes raised by the reviewer:

  1. Pure DOA (signal-level, no model) reaches 53.0% indoors (96% in the low-reverberation outdoor control) — signal-level spatial cues do exist in the four-channel recordings, degraded by reverberation;
  2. MLLM-alone is near chance (AV 9.8%–14.4%, AO 12.9%) — consistent with the mono-input evidence above: inter-channel cues are stripped before reaching the model;
  3. DOA→MLLM (text hint) recovers to 46.2%–53.0%, matching pure DOA — once the cue arrives in text form, models use it;
  4. Oracle DOA→MLLM reaches 81.8%–94.7% — models have ample headroom in consuming correct spatial cues.

Levels 3–4 show that models can exploit spatial cues supplied as text, identifying reliable cue acquisition as the bottleneck under the evaluated mono interfaces. However, interface stripping and model-level spatial-cue extraction remain confounded. The revision will therefore scope our spatial conclusions to "models via their actual interfaces" and list natively multi-channel models as future work (Appendix H). Native channel-preserving evaluation is needed to isolate the two causes; if useful, we can run an endpoint calibration with known-direction panned stimuli.

3. DOA method and array details.

DOA estimator. GCC-PHAT with an SRP-PHAT search: 50 ms frames, 25 ms hop, 16 kHz; per frame, PHAT-weighted generalized cross-correlation over all six microphone pairs (nfft = 16384), scanning azimuths 0°–359° in 1° steps and summing correlations at the theoretical pair delays; the argmax gives the frame-level estimate. Frames below 1.5× the segment mean energy are discarded (energy VAD on one reference channel), and the clip-level estimate is the modal 10° histogram-bin center over surviving frames (more robust to onset transients than a median).

Array geometry. Four microphones at (±33 mm, ±33 mm) relative to the array center — a square of 66 mm side (93 mm diagonal; max inter-channel delay ≈ 4.3 samples at 16 kHz) — from the official XVF3800 AEC_MIC_ARRAY_GEO. The x-axis points right, y-axis forward. Channel mapping (0-based): ch2→rear-right, ch3→rear-left, ch4→front-left, ch5→front-right. Azimuth is measured clockwise from the front (front = 0°, right = 90°, …, front-left = 315°).

Hint-generation window. The DOA hint is computed over the full 60 s clip (trailing segments < 3 s discarded). The direction recordings are controlled: each session contains a single source at a fixed, known azimuth (recorded in the session metadata), so the whole-clip mode is a stable estimate with no need for event-onset alignment.

We thank the reviewer again for the careful reading and constructive comments, which have genuinely helped us improve the accuracy of the paper's presentation.

AuthorsOpenReview ↗

Response to Q2: Matched cascade evaluation (predicted vs. gold premise) and the blind-rule prior baseline (complete results)

We sincerely thank the reviewer for this key question. The supplementary no-gating evaluation is now complete for all three models (Q0 is answered and scored as usual, but a wrong Q0 no longer skips later questions; every event runs through its full chain; 20 sessions per model, 132 events = 84 simple + 48 complex). The earlier access difficulties caused by Google's phase-out of Gemini-2.5-Pro have been resolved. The two comparisons follow.

(1) Matched cascade evaluation: predicted vs. gold premise, same clips, same denominator

On exactly the same 48 complex clips, the downstream closed-form items are fixed at 2 per clip (CQ2 priority ranking + CQ5 action decision), giving a fixed denominator of 96 items. Under the predicted premise, CQ2's options are dynamically generated from the sounds the model itself selected in CQ1, and CQ5's stem and gold answer follow the model's CQ2 choice; if CQ1 selects fewer than 2 sounds so that no branch can be generated, the clip's 2 items score 0 with the denominator kept. Under the gold premise, ground-truth event pairs are written directly into the stem.

ModelPredicted premise (96 items)Gold premise (96 items)
Qwen2.5-Omni-7B93/96 = 96.9%96/96 = 100%
VideoLLaMA2.1-7B-AV44/96 = 45.8%96/96 = 100%
Gemini-2.5-Pro95/96 = 99.0%96/96 = 100%

VideoLLaMA2's 45.8% decomposes completely: on the 22 clips that actually branched, its CQ2+CQ5 is 44/44 = 100%; all 52 lost items come from the 26 clips where CQ1 selected fewer than 2 sounds and no branch could be generated. Gemini's single lost item sits in a clip whose CQ1 was already partially wrong (0.5 points), so its loss likewise originates on the perception side. The entire gap between the two premises comes from the branchability of the perception output, not from ranking or decision ability.

(2) After accounting for the smoke-alarm and majority event-pair priors

We constructed the blind-rule baseline the reviewer refers to (no audio-visual input; fixed rules: Q0 answer "yes"; CQ1 answer "smoke alarm + knock"; CQ2 answer "smoke alarm"; CQ3 random direction; CQ4 answer the majority scene item; CQ5 answer "evacuate"), scored under the official rules on the same 48 clips and 288 item slots:

Q0CQ1CQ2CQ3CQ4CQ5Chain total
Blind-rule baseline100%84.4%100%≈12.5%–18.8%100%100%≈82.8%–83.9%
Qwen2.5-Omni-7B97.9%17.7%93.8%21.9%97.9%100%71.5%
VideoLLaMA2.1-7B-AV97.9%64.6%45.8%16.7%45.8%45.8%52.8%
Gemini-2.5-Pro87.5%88.5%97.9%17.7%100%100%81.9%

(VideoLLaMA2's identical 45.8% on CQ2/CQ4/CQ5 comes from the 26 unbranchable clips scoring 0 on all three items. The baseline's CQ4 = 100% is by design — that item is a scene-understanding question whose gold answer is constant across clips and deliberately unambiguous, serving stage isolation.)

What changes and what does not:

  • "Perception is the bottleneck": unchanged, and strengthened. Both open-source models score below a zero-perception blind-rule baseline — their failure is not merely "weak perception" but perception-side behavior actively losing points (Qwen's CQ1 select-all triggering the zero-score rule, 17.7% vs. the blind 84.4%; VideoLLaMA2's under-selection leaving 26 clips unbranchable). Gemini is the only model exceeding the blind baseline on CQ1 (88.5% vs. 84.4%; exact match 79.2% vs. 68.8%). No model exceeds the blind baseline in chain total. The score differences are localized to perception and its downstream branching consequences: Q0 and CQ1 vary across models, CQ3 is near-random, and VideoLLaMA2's 45.8% on CQ2/CQ4/CQ5 results from the 26 unbranchable clips rather than independent downstream failures. Thus, perception errors and their branching consequences dominate end-to-end loss.
  • "Downstream is close to solved": the claim is narrowed. The reviewer's criticism holds — under the current event composition (all 48 clips contain the smoke alarm, with a fixed priority rule and action mapping), high downstream scores are achievable by a blind rule. These numbers should therefore be read as "stage decoupling works, and downstream adds no loss beyond the rule-achievable baseline when perception is correct", not as evidence of robust ranking/decision ability. We will tighten the §5.3 wording accordingly and state this scope explicitly.
  • Follow-up improvement: the event-composition imbalance is already listed as a limitation in §6; the next data release will balance event pairs (reducing the smoke-alarm share) so that the blind event-pair strategy no longer works and the downstream items regain discriminative power. We will also add the blind-rule cascade baseline to the revision as a fixed reference.

We thank the reviewer again — this question prompted us to run these experiments, completing the evidence chain behind our attribution analysis.

AuthorsOpenReview ↗

Response to Q3: Direction annotation, verification, and the source of the 132 doorbell clips

We sincerely thank the reviewer for raising this point, and we apologize for the insufficient clarity in the manuscript. We explain honestly and provide additional data below.

1. How event directions were annotated. Directions are head-relative bearings, recorded over each event's time span with one annotation at the event's onset and one at its end. The direction question (Q2) asks from which direction the sound initially came, and the label semantics are aligned with the question's definition — we evaluate static localization at the event onset, not continuous bearing tracking.

2. An honest statement of the limitation (head motion). The reviewer's concern is valid: the wearer may turn their head within an event window, and the two onset/offset annotations cannot rule out mid-event changes. We have not systematically verified head motion during events, and this is the boundary of the current annotation scheme. The recording device carries an IMU that continuously logs head pose; these data will be released with the dataset. Should dynamic bearing tracking be needed, we will build an advanced annotation scheme with continuous head-motion calibration.

3. DOA method and array calibration. The DOA estimator is GCC-PHAT with an SRP-PHAT search (50 ms frames, 25 ms hop, six microphone pairs, 1° azimuth scan, energy VAD with modal 10° histogram aggregation). The array geometry is taken from the official XVF3800 AEC_MIC_ARRAY_GEO — four microphones at (±33 mm, ±33 mm), a 66 mm square, max inter-channel delay ≈ 4.3 samples at 16 kHz — and this official specification serves as the basis for array calibration. Full estimator parameters, channel mapping, and azimuth conventions are given in our response to Question 1.

4. The source of the 132 doorbell clips and their relationship to the core dataset. The 132 doorbell clips used in the DOA intervention (Table 7) come from an independent direction-recording dataset, collected specifically for the controlled direction intervention and separate from the six core scenarios. The eight directions are covered in a balanced way: front 16, front-right 18, right 16, rear-right 16, rear 17, rear-left 16, left 17, front-left 16 (12.1%–13.6% per direction, close to uniform). Each session contains a single source at a fixed, known azimuth (recorded per session in the metadata), so stable direction ground truth is available without event-onset alignment.

5. Verification: per-direction model behavior and a human baseline. On this balanced dataset, all three models under MLLM-alone remain close to the 12.5% random baseline overall (Qwen 14.4% AV, Gemini 7.6% AV, VideoLLaMA 9.8% AV; all 12.9% under AO). The per-direction breakdown further reveals a consistent failure pattern: each model's few correct answers concentrate on one or two fixed directions (Qwen on "front/rear," Gemini on "front-left," VideoLLaMA on "front"), with nearly all remaining directions wrong — a directional response prior rather than localization ability (the full per-direction table is in our response to Question 4). In addition, we measured a human baseline on the direction task: 10 participants, 100 items in total, 92% overall accuracy, no direction below 76.9%, with errors concentrating on rear directions — consistent with the known difficulty pattern of human auditory localization. The task and its labels are fully solvable for humans, which further corroborates the reliability of the direction labels.

We thank the reviewer again for the careful reading — the suggestion has genuinely made the evidence chain behind our direction-related conclusions more complete.

Reviewer_3G3ROpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

EgoSPECT introduces an egocentric audio-visual benchmark with roughly 60 hours of recordings across six scenarios. It evaluates three MLLMs through linked perception, priority, and decision questions, with human comparisons and modality controls. The main finding is that perception dominates errors under the proposed protocol.

Critical strengths

The wearable recordings and negative controls cover a useful evaluation setting. Human comparisons, blind-prior baselines, and option-shuffle controls expose dataset and prompt effects that aggregate accuracy alone would hide.

Critical weaknesses
  1. Limited novelty. EgoSound (Zhu et al., 2026) and SAVVY-Bench (Chen et al., 2025) already evaluate egocentric sound understanding and spatial reasoning. All 48 complex clips contain a smoke alarm; a blind event-pair predictor achieves 68.8% exact match; and all models solve the 96 gold-premise downstream items (Section 5.5). The added stages are therefore too narrow to establish a substantial advance in evaluating decision-making.
  2. The bottleneck attribution is not cleanly established. Q0 errors zero all subsequent items, while downstream answers can be graded against the model’s own incorrect event premise (Section 3.4 and Appendix F). The end-to-end/conditional gap therefore partly reflects the scoring design and does not directly measure successful decisions about the actual scene. Spatial failures are also difficult to interpret when Qwen’s pipeline downmixes audio to mono (Section 4), removing inter-channel cues. These confounds weaken the central diagnostic claim.
  3. Poor writing and presentation. The writing and presentation need substantial improvement. The overview figure is overloaded with text, the labels in Figure 2 are too small, and Figure 3 occupies excessive space. These issues obscure the benchmark design and make the paper unnecessarily difficult to follow.
Questions for authors
  1. Clarification: How many distinct priority/action templates are used, and how well does a fixed event-to-action rule perform on them?
  2. Requested validation: Does the bottleneck conclusion hold when final decisions are scored against the true scene on matched clips, and spatial evaluation preserves audio channels?
5 comments
AuthorsOpenReview ↗

Response (Part 1 of 2): Novelty

Thank you very much for your careful review of my manuscript and for your valuable comments. This part responds to your point on novelty.

1. On novelty. Our contribution is not another egocentric sound benchmark, but a cascaded, stage-decoupled evaluation protocol (premise externalization, dynamic branching, controlled probes) linking perception, priority, and decision with stage-level error attribution. Table 1 (in §2 Related Work; included as Table 4 in Appendix A of the original submission) compares 26 benchmarks dimension by dimension (✓ = covered, partial = limited, — = absent):

BenchmarkEgoSpat. dir.Event sem.Multi-ev. priorityAction decisionNeg. controlsStage-wise diagnosis
EgoSound✓✓✓————
EgoAVU✓partial✓——partial—
SAVVY-Bench✓✓partial————
SoundSpaces / 2.0partial✓——partial——
BeDAViNpartial✓partialpartialpartial——
STARSS23—✓✓partial———
EgoCom / EasyCom✓partialpartial————
Ego4D / EPIC-KITCHENS✓partial/—✓————
HoloAssist / EgoLife / EgoThink / EgoBlind / EgoSAT✓—✓/partial—partial——
SAW-Bench✓partialpartial—partial——
MMAU / AudioBench——✓—partial/———
EgoSPECT (ours)✓✓✓✓✓✓✓

As shown, both cited benchmarks lack priority, decision, negative controls, and stage-wise diagnosis. The comparison is now presented in §2 (Table 1) and is explicitly cited in the introduction.

To be explicit about the difference: EgoSound and SAVVY are capability benchmarks evaluating whether a model can recognize sounds or localize a source, whereas EgoSPECT is a diagnostic benchmark designed to identify where and why the perception-to-priority-to-decision chain fails. Beyond the evaluation protocol, the underlying task settings also differ:

  • Target level: SAVVY focuses on 3D spatial relations of sounding objects in controlled multi-room apartments (indoor scenes); EgoSPECT targets safety-critical environmental events (knocks, alarms, vehicles) requiring semantic priority ranking.
  • Acoustic complexity: EgoSound focuses on open-ended sound understanding and explicitly filters out "excessive background noise" (Sec 3.1). In contrast, EgoSPECT treats high-noise transit, outdoor traffic, and reverberant rooms as core testbeds.
  • Controls & Infrastructure: Neither EgoSound nor SAVVY evaluates event priority, behavioral decisions, nor negative controls. Furthermore, to rigorously study complex embodied perception, we built a custom headset and integrated annotation software that enables a hybrid annotation paradigm. Unlike purely post-hoc pipelines used in EgoSound, SAVVY, and other previous benchmarks (which rely on retrospective video review and suffer from hindsight bias), our system supports first-hand, on-the-go labeling to capture the exact sensory state the agent reacts to, while allowing subsequent revision of ground truth data.

To clarify the two data points you mentioned: (i) the blind-majority prior (68.8% exact) is our own baseline (Section 5.5) for detecting prior-driven guessing — Gemini's 95.8% exceeds it, evidencing genuine multi-event perception; the imbalance is acknowledged in Section 6. (ii) 100% gold-premise accuracy is the protocol's intended diagnostic: supplying the true event pair isolates downstream competence and attributes the end-to-end loss to perception (Section 5.3 notes open-ended decision-making is not thereby solved). These results show that the stages are separable, which is the protocol's goal, rather than that the additional stages are trivial.

AuthorsOpenReview ↗

Response (Part 2 of 2): Bottleneck Attribution, Question Design, and Presentation

This part responds to your points on bottleneck attribution, question templates, and presentation.

2. On bottleneck attribution. The three evaluation lenses are complementary: end-to-end scores against the true scene (a wrong Q0 zeroes the chain), conditional accuracy isolates Q0 gating, and gold-premise scores downstream items against the true premise; the ~30-point gap is attributed to Q0 gating (Section 5.3). On channels: VideoLLaMA consumes only the first channel, and Qwen and Gemini downmix to mono (our Section 4 statement on Gemini was incorrect; we apologize and will correct it in the revision). We agree that the model's inability to extract spatial cues and the interface stripping them are currently confounded, and we will note this as a limitation. The diagnosis holds under either cause: Table 7 shows spatial cues exist at the signal level (pure DOA 53%) and, once provided (Oracle DOA 82–95%), are used well — the bottleneck lies in acquiring spatial cues.

Importantly, our custom headset provides a fully transparent hardware design with a known microphone array geometry and synchronized raw channels. While commercial smart glasses (e.g., Aria) provide data export, their internal acoustic preprocessing and factory calibration remain opaque. Our open design, in contrast, enables an independent, physical DOA calibration and a fully reproducible channel-preserving audit, which we will include in the revision. Moreover, analogous to open-source hardware platforms like Crazyflie in robotics, we will publicly release our headset design, BOM, and annotation tool to enable the community to reproducibly collect complex embodied audio-visual data.

3. On the questions. (a) Templates: We use 1 priority template (CQ2, dynamic options; merged into the decision item in basic scenarios, Section 3.3) and 4 decision template families (basic Q4, complex CQ5, subway behavioral items, traffic Q5 — one template per clip). Actions follow a deterministic event-to-action mapping (3 indoor rules plus subway/traffic rules, Appendix B). To answer directly: given a correct premise, a fixed rule reaches 100%, consistent with all three models solving the 96 gold-premise items. Decision items are kept unambiguous so each stage measures only itself; our goal is localizing the bottleneck, not maximizing decision difficulty. We will report this fixed-rule baseline in the revision; richer decision tasks are future work. (b) The first half of your question is already covered by the gold-premise evaluation: these are text-only items whose stems state the ground-truth event pair, with no audio/video, and decisions are scored against the true scene; all 96 are answered correctly by all models. For the second half, we must honestly state that it cannot be verified at present, as the evaluated model interfaces currently collapse to mono. Our conclusions are scoped to the models' actual deployment interfaces, and the revision will state this explicitly. Channel-preserving spatial evaluation is future work.

4. On presentation. We appreciate these concrete suggestions and will declutter Figure 1, enlarge Figure 2 labels, compact Figure 3, and do a full language pass.

We thank you again for the careful reading and constructive comments, which have genuinely helped us improve the accuracy of the paper's presentation.

Thank the authors for the detailed response which addressed my concerns partially but the limitations such as all 48 complex clips containing a smoke alarm and the model's inability to extract spatial cues are still there. I will raise my rating to 2.

Reviewer_kKxuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a benchmark for first person event-driven embodied intelligence which decomposes the problem into perception, priority assessment and decision making stages. The argument made here is that existing benchmarks test each of these components in isolation while next action decisions in real life use cases with wearable devices involve all the stages from perception to decision making. The work presents experimental proof that perception is currently the biggest bottleneck in these systems with spatial localization being the hardest challenge for current omni-modal models.

Critical strengths
  • The work is well motivated with a clear explanation of the current evaluation landscape and the need for end-to-end testing.
  • The benchmark design including the hardware used and the sub-set of task categories explored is sufficient.
  • The control ablations are well designed to isolate error patterns to the stages where they occur.
Critical weaknesses
  • The 4 condition evaluation claim in the abstract is slightly over stated. While Audio-only and Audio-Visual conditions are well explored, Video-only condition is missing and text-only condition is mostly just a control experiment.

  • The design isolates audio and visual inputs to the perception stage only and the remaining 2 stages use a text premise that is derived by the perception responses. While this is great for diagnosis, the effect of injecting sensory inputs into the priority and decision making stages is not explored.

Questions for authors
  • In Table 1, is the AV or the AO input condition used and can the label be updated accordingly?
  • The abstract specifies a video only input conditions. But I was not able to find results for the same. Can you point me to where they are reported or clarify that the condition was not evaluated?
4 comments
AuthorsOpenReview ↗

Response: Table 1 labeling (AV) & video-only condition results

We sincerely thank the reviewer for the careful reading. Both questions point to places where our writing was imprecise, and we take them seriously.

Q1: In Table 3, is the input condition AV or AO?

Thank you for catching this — it is our oversight. Both the End-to-end and the Conditional results in Table 3 were obtained under the AV (audiovisual) condition. We will explicitly label the AV condition in the table and its caption in the revision. Note that this is purely a labeling omission and does not affect any of the reported numbers or conclusions.

Q2: The abstract mentions a video-only (V) condition, but its results cannot be found.

Thank you for pointing this out. We did run the V condition in the basic scenarios (knocking, smoke alarm, doorbell); the per-scenario condition settings are listed in Appendix C, Table 10. However, because EgoSPECT's question chains are built around sound events — Q1 (perception), Q2 (direction), and Q4 (action) all rely on auditory information — the full question chain is structurally unanswerable under video-only input; only the wearer-context question (Q3) can be answered. V therefore served only as a supplementary control in the basic setting, and its results were not separately reported in the main text. As a consequence, the abstract and conclusion characterize the "four input conditions" more broadly than our experiments support — an imprecision on our side. We will revise the abstract and conclusion to state explicitly that V serves only as a supplementary control in the basic setting.

We also take this opportunity to report, for the first time, the complete V-condition results: across the three basic scenarios, the three models achieve an overall accuracy of 99.6% (481/483) on Q3. The per-model, per-scenario breakdown is as follows:

ModelKnockingSmoke alarmDoorbellTotal
Gemini68/68 (100%)49/50 (98%)43/43 (100%)160/161 (99.4%)
Qwen68/68 (100%)49/50 (98%)43/43 (100%)160/161 (99.4%)
VideoLLaMA268/68 (100%)50/50 (100%)43/43 (100%)161/161 (100%)
Total204/204 (100%)148/150 (98.7%)129/129 (100%)481/483 (99.6%)

This shows that when a question is answerable from the video channel, the models exploit visual information well; the incomplete chain under V is structural rather than a model failure. It is also consistent with the paper's central claim — the bottleneck lies in the perception and spatial localization of sound events, not in extracting information from the visual channel. We will add these results to Appendix C in the revision.

We thank the reviewer again for the careful reading and constructive comments, which have genuinely helped us improve the accuracy of the paper's presentation.

AuthorsOpenReview ↗

Rationale for Premise Externalization and Sensory Injection into Downstream Stages

We thank the reviewer for raising this important point. We agree that directly injecting audiovisual sensory input into the priority and decision-making stages, and systematically examining its effects, is a valuable research direction.

Indeed, in our initial experiments, we used a setup in which audiovisual inputs were provided to both the upstream perception stage and the downstream priority/decision stages. However, inspection of the models’ reasoning processes revealed that this setup led the models to perceive the same event twice in a relatively independent manner, and the two perceptual interpretations could be inconsistent. For example, a sound recognized as a doorbell in the upstream perception stage could be reinterpreted as a smoke alarm when perceived again downstream. In such cases, it becomes difficult to determine whether a downstream error stems from the priority/decision-making process itself or from a perceptual shift introduced by this second round of perception.

We therefore deliberately externalize the upstream perception output as a textual premise in our current design, so that the downstream priority and decision stages operate on the same premise. This prevents the second round of perception from introducing additional errors and enables stage-decoupled diagnosis across perception, prioritization, and decision-making. In other words, this design does not disregard the effects of sensory input on downstream stages; rather, it is intended to ensure clear stage-level attribution.

The reviewer is correct that the effect of downstream sensory injection itself has not been systematically explored in the current work. We identify this as a clear direction for future work: comparing shared textual premises with renewed audiovisual injection at downstream stages, and analyzing how upstream–downstream perceptual consistency affects final prioritization and decisions.

Thank you for the detailed responses. Table 3 is updated with the condition used. V is now rephrased as a supplementary control and V-condition results where visual cues are sufficient for the task are provided. This resolves my questions.

The rationale for the design choice of not injecting audio visual inputs to the downstream stages is convincing. I still think that the paper would be stronger with an ablation on a subset comparing the shared text premise with AV injection in downstream stages.

All my main concerns have been addressed and the revised paper is more accurate. While the diagnostic design and control experiments are valuable, the headline results are still partially limited by the design choices. I am maintaining my score of 3.

AuthorsOpenReview ↗

No-Gating, Dynamic Branching, and the Downstream AV-Injection Ablation

We sincerely thank the reviewer for suggesting a controlled comparison between shared textual premises and downstream AV injection. We have completed this experiment for Qwen, VideoLLaMA2, and Gemini and first clarify the protocol below.

No-gating and dynamic branching. No-gating removes the procedural gate at Q0: regardless of the Q0 answer, even if incorrect, the model proceeds to subsequent questions. Thus, Q0 no longer affects downstream question coverage. However, no-gating does not remove the CQ1-based branching mechanism:

  1. In CQ1, the model selects the sound events it perceives.
  2. If at least two sounds are selected, CQ2 is generated with those sounds as options.
  3. CQ5 then uses the event selected by the model in CQ2 as its explicit action-selection premise.
  4. If fewer than two sounds are selected, the corresponding CQ2/CQ5 branch cannot be generated.

Thus, no-gating removes the Q0 gate, but not the CQ1-based branching requirement.

Ablation protocol. The control retained the original text-only CQ2/CQ5. The AV condition added the original audio-video input to CQ2/CQ5 while preserving the textual premises. All other questions, branching rules, and scoring procedures were unchanged.

Descriptive results over all generated branches
ModelMeasureTextAV injection
QwenCQ245/4845/48
QwenCQ548/4848/48
QwenCQ2+CQ593/9693/96
VideoLLaMA2Clips with generated branches2227
VideoLLaMA2CQ2, generated branches22/2227/27
VideoLLaMA2CQ5, generated branches22/2227/27
VideoLLaMA2CQ2+CQ5, generated branches44/4454/54
VideoLLaMA2CQ2+CQ5, fixed 96 slots44/9654/96
GeminiCQ2+CQ595/9695/96

For VideoLLaMA2, the increase from 44/44 to 54/54 primarily reflects more generated branches (22→27 clips), not improved accuracy on the same downstream questions. Its CQ1 score changed from 31/48 to 30/48, so the additional branches also cannot simply be interpreted as more accurate perception.

Matched comparison with identical textual premises

Because CQ2 options and CQ5 premises are dynamically generated from previous answers, a change in CQ1 changes the downstream questions themselves. To isolate AV injection on the same questions, we selected clips that generated branches in both runs and had exactly identical CQ1 answers. For VideoLLaMA2, 18 clips met these criteria; each contributes one CQ2 and one CQ5, yielding 36 questions.

ModelMatched subsetCQ2: Text→AVCQ5: Text→AVTotal: Text→AV
Qwen48 clips, identical CQ145/48→45/4848/48→48/4893/96→93/96
VideoLLaMA218 clips, identical branches and CQ118/18→18/1818/18→18/1836/36→36/36
Gemini45 clips, identical branches and CQ144/45→44/4545/45→45/4589/90→89/90

On these matched questions, AV injection produced no observable change in CQ2/CQ5 accuracy. Thus, textual externalization itself caused no observable score loss on these closed-form downstream questions, and VideoLLaMA2’s aggregate increase should not be attributed to improved downstream reasoning from AV injection.

We acknowledge that these questions were already near or at ceiling in the text-only condition, limiting sensitivity to subtler downstream differences. We therefore do not claim that AV is generally unhelpful; our conclusion is limited to the current tasks, interfaces, and question design.

The Gemini results are now included and do not change these conclusions. We will also add these conclusions to the main paper and provide a detailed explanation of the protocol and matched-subset analysis in the appendix.

We thank the reviewer again for this constructive suggestion, which has helped us distinguish upstream branching changes from downstream AV-injection effects and more precisely scope our conclusions.

Comments on the paper

13 comments
AuthorsOpenReview ↗

Response to Weakness 4 (Part 1 of 2): Score discrepancies and a controlled retest of VideoLLaMA2 traffic decisions

We sincerely thank the reviewer for carefully cross-checking our numbers. The inconsistencies identified here are real reporting flaws on our side — the figures come from different item sets and premise settings that the manuscript failed to state — and we are grateful for the chance to correct them. (Part 1 of 2: the score discrepancies; scoring rules, exclusion accounting, the item-level ledger, and session-level uncertainty follow in Part 2.)

1. Spatial scores: 25.4% (Fig. 4) vs. 21.9%–23.5% (§5.5) — different item sets. The 21.9%–23.5% range covers the eight-direction items (Q2) of the basic scenarios only; the spatial-direction dimension in Figure 4 aggregates direction items from both basic and complex scenarios, under which Qwen scores 25.4%. The revision will state the item scope of each number in the Figure 4 caption and §5.5 and unify the reporting scope.

2. Traffic scores: 100% (284/284) vs. 79.2% — different scopes, and a deeper problem found on retest. 284/284 is the cascade scope: Q5 takes the model's own Q4 answer as premise with dynamically generated gold labels, and 15 "cannot determine" premises are skipped per protocol, hence 284 of 299 items. 79.2% is the §5.5 option-robustness probe with gold premises on all 299 items. The two differ in premise source and denominator.

While verifying these numbers we found the VideoLLaMA2 evaluation configuration had malfunctioned (empty outputs on re-run). We confirmed that weights, environment, code, and drivers were unchanged, localized the fault to output generation, and repaired it under the original configuration. We therefore treat VideoLLaMA2 traffic Q4/Q5 as a complete fresh retest (seed=42; premise source × option shuffling; per-item logging) and will update the paper accordingly.

Table A: accuracy

Cond.PremiseQ4 opt.Q5 opt.Q4 acc.Q5 acc.
A1ownfixedfixed35.8% (107/299)52.7% (148/281)
A2ownshuffledfixed49.2% (147/299)41.6% (123/296)
A3ownfixedshuffled35.8% (107/299)67.3% (189/281)
A4ownshuffledshuffled49.2% (147/299)44.6% (132/296)
B1gold—fixed—59.9% (179/299)
B2gold—shuffled—72.2% (216/299)

("Cannot determine" Q4 premises skip Q5 per the cascade protocol: 18 skipped in A1/A3, 3 in A2/A4 — consistent with the 284 scope. The original 79.2% will be replaced by B2 = 72.2%.)

Table B: Q4 option distribution (A3/A4 rows identical to A1/A2 and omitted)

Cond.nOut of viewIn viewNo vehicleCannot det.
A1299148 (49.5%)98 (32.8%)35 (11.7%)17 (5.7%)
A2299123 (41.1%)143 (47.8%)30 (10.0%)3 (1.0%)
Gold299179 (59.9%)120 (40.1%)00

(A1 also has 1 empty response.)

Table C: Q5 option distribution (content / letters)

Cond.nMind behindWatch aheadWalk onA/B/C
A1281281 (100%)00281/0/0
A2296296 (100%)00296/0/0
A3281236 (84.0%)39 (13.9%)6 (2.1%)126/77/78
A4296213 (72.0%)79 (26.7%)4 (1.4%)169/57/70
B1299299 (100%)00299/0/0
B2299262 (87.6%)37 (12.4%)0123/88/88

Findings: (i) Content-level mapping collapse in the decision layer. Under all Q5 fixed-order conditions, all 876/876 answers are "mind vehicles behind." The original 100% was an illusion: perception also collapsed (279/299 Q4 answers "out of view"), whose dynamic gold label happens to be "mind vehicles behind." B1's 59.9% exactly equals the "out of view" share of premises (in-view: 0/120; out-of-view: 179/179). (ii) Content-level, not position-binding. After shuffling, letters spread (A: 41%–57%) but content stays 72%–88% on "mind behind." (iii) Shuffling helps in the right direction. Shifts toward "watch ahead" match premise needs (all 37 such answers in B2 correct); Q5 accuracy rises 52.7%→67.3% (A1→A3) and 59.9%→72.2% (B1→B2). (iv) Position prior in perception. Fixed-order Q4 leans "out of view" (49.5%); shuffling shifts toward "in view" (47.8%), drops "cannot determine" from 17 to 3, and raises Q4 accuracy 35.8%→49.2%.

In the revision, Table 15 / Figure 4 will use the retest results with premise source and denominator stated; §5.5 will present the full control matrix; and the conclusion will be corrected from "100% traffic decision accuracy" to a content-level premise-to-action mapping collapse previously masked by the perception collapse. We thank the reviewer for prompting this re-examination — the corrected result is a clearer failure-mode finding for cascade evaluation.

We thank the reviewer again for the careful reading and constructive comments, which have genuinely helped us improve the accuracy of the paper's presentation.

AuthorsOpenReview ↗

Response to Weakness 4 (Part 2 of 2): Scoring rules, exclusion accounting, item-level ledger, and session-level uncertainty

We sincerely thank the reviewer again. (Part 2 of 2: scoring rules, exclusion accounting, the item-level score ledger, and session-level uncertainty. Part 1 addresses the spatial and traffic score discrepancies.)

3. CQ1 partial-credit rule (unchanged since the first version; will be made explicit in Appendix E). CQ1 is a multi-select item with three options (knocking / doorbell / smoke alarm; ground truth: the 2 events that actually occurred): each correctly selected option earns 0.5 points, up to 1.0 (both correct = 1.0; exactly one correct = 0.5); selecting all three options scores 0 directly, preventing a select-all gaming strategy; wrong selections incur no additional penalty.

4. How incomplete answers enter the final score (scenario-specific pre-specified protocols). Complex / priority scenario: an answer with an unparseable option letter scores 0 with the denominator retained — we note that this case never actually occurred in any official run — and when CQ1 selects fewer than 2 sounds so that no downstream branch can be generated, the clip's CQ2–CQ5 items (4 items) score 0 with the denominator retained, while Q0 and CQ1 itself are scored normally; a wrong CQ1 with ≥2 selections receives partial credit and downstream items follow the model's own branch, without guilt by association. Traffic scenario: the very few items that did not receive complete answers under extreme conditions are excluded per Table 10 footnote ‡ (e.g., 783 items scored for Qwen, 769 for VideoLLaMA2), the only source of model-dependent denominators.

5. Repeated decoding runs. Within each evaluation protocol, each (question × modality condition) combination is decoded exactly once; local models use greedy decoding (do_sample=False), Gemini uses temperature=0.0, and its at-most-3 retries only handle API transport/service errors with unchanged input, not additional decoding runs. AV and AO are independent evaluation conditions, not repeated runs, and no repeated decodes are averaged within any reported value. Main-run, supplementary no-gating, and explicitly labeled retest results are reported separately. Figure 3's Complex score combines disjoint item sets from the main evaluation and the separately labeled Q0/CQ1 perception retest, rather than repeated answers to the same items.

6. Item-level score ledger and session-level uncertainty. The complete item-level scoring records will be released on the project website together with the dataset upon publication — one row per question with session/clip identifiers, qid, modality condition, raw response, parsed option, score, and exclusion/skip flags (with reasons) — plus a mapping between each reporting scope (end-to-end / conditional / gold-premise / each Figure 4 dimension) and its item set.

For uncertainty, questions within a recording session share the environment, acoustics, and wearer, so their scores are not independent samples. We use a session-level cluster bootstrap: average item scores within each session, resample sessions with replacement 10,000 times, and take the 2.5%/97.5% percentiles of the bootstrap means as the 95% CI. Results under the AV condition (full table in the revision):

ScenarioSessions/ItemsGeminiQwenVideoLLaMA2
Direction (doorbell, MLLM-only)8 / 1327.7% [0.0, 20.2]14.3% [3.1, 26.7]9.9% [2.9, 19.2]
Metro17 / 17969.6% [61.3, 78.2]49.5% [43.4, 57.4]50.3% [37.7, 62.3]
Traffic13 / 45866.9% [59.3, 74.7]44.5% [33.8, 53.7]41.7% [31.9, 50.5]
Priority (complex)20 / 55975.7% [72.5, 78.8]53.5% [42.1, 63.4]54.0% [41.7, 64.8]

Interpretation: (1) with session correlation accounted for, Gemini's AV intervals are separated from both alternatives on Traffic and Priority; Gemini has the highest point estimate on Metro, but its interval overlaps VideoLLaMA2's; on Direction, Qwen has the highest point estimate and all intervals overlap. In the full table, Gemini's intervals are also separated on Basic AV; (2) Qwen and VideoLLaMA2 show larger session-to-session variance (session SD 0.15–0.27, individual sessions as low as 0%); (3) on the direction task all three CIs cover the 12.5% random baseline, consistent with the near-chance localization finding; (4) some supplementary conditions have few sessions (e.g., traffic audio_only: 9 sessions, 27 items) and wide intervals, which we mark with session/item counts in the appendix. Per-session score data will be released upon publication.

We thank the reviewer again for the careful reading and constructive comments, which have genuinely helped us improve the accuracy of the paper's presentation.

AuthorsOpenReview ↗

Response to Q1: Channels reaching each model after preprocessing, and DOA estimator details

We sincerely thank the reviewer for the careful reading and for this precise question. It points to a place where our manuscript was genuinely imprecise, and we take it seriously.

Which channels reach each model after preprocessing?

Upon verification, all three evaluated models discard inter-channel information at the input stage — they just discard it in different ways:

ModelProcessingSignal reaching the model
Qwen2.5-Omni-7Blibrosa.load(mono=True): channels averaged16 kHz mono (average of 4 mics)
Gemini-2.5-ProChannels combined on the API sideMono (per official documentation)
VideoLLaMA2.1-7B-AVwav[:, 0]: only channel 0 kept16 kHz mono (a single microphone)

The specific evidence:

  • Qwen2.5-Omni-7B (downmixed to mono). The official toolchain qwen-omni-utils loads audio with librosa.load(sr=16000), whose mono parameter defaults to True, averaging all channels into one waveform. Passing a multi-dimensional numpy array directly raises ValueError("Support only mono audio"), so the official interface does not support multi-channel input at all. (Code: qwen-omni-utils/src/qwen_omni_utils/v2_5/audio_process.py in the Qwen2.5-Omni repository.)
  • VideoLLaMA2.1-7B-AV (channel 0 only — not a downmix). Both audio front-end paths keep a single channel: a directly read wav (N, C) is reduced to wav[:, 0], and audio extracted from video is forced to mono via ffmpeg -ac 1. The signal is then resampled to 16 kHz and converted to 128-dim Kaldi fbank features for the BEATs encoder, at which point inter-channel time/level differences are unrecoverable. (Code: videollama2/mm_utils.py at commit 8cedfbd.)
  • Gemini-2.5-Pro (combined into one channel). Google's official Audio understanding documentation states: "If the audio source contains multiple channels, Gemini combines those channels into a single channel."

Two statements in §4 of the original manuscript are therefore incorrect — "we feed the same four-channel raw microphone audio to every model" and "Gemini receives the raw channels natively" — and the revision will correct them. We therefore scope the finding to the evaluated deployment interfaces: all three models receive mono input, and the laddered results identify acquisition of a reliable spatial cue as a key bottleneck. DOA hints recover accuracy to ~50% and Oracle DOA reaches 81.8%–94.7%, showing that the models can exploit spatial cues when these are supplied explicitly as text. However, interface stripping and model-level spatial-cue extraction remain confounded; native channel-preserving evaluation is required to separate them. If the reviewer considers it necessary, we can additionally run an endpoint calibration test with known-direction panned stimuli to directly verify what each endpoint preserves.

DOA estimator. GCC-PHAT with an SRP-PHAT search. Each clip is analyzed in 50 ms frames with a 25 ms hop at 16 kHz. Per frame, we compute the PHAT-weighted generalized cross-correlation of all six microphone pairs (nfft = 16384) and scan azimuths 0°–359° in 1° steps, summing correlations at the theoretical pair delays; the argmax gives the frame-level estimate. Frames below 1.5× the segment mean energy are discarded (energy VAD on one reference channel), and the clip-level estimate is the center of the modal 10° histogram bin over surviving frames, which we found more robust to onset transients than a median.

Array geometry. Four microphones at (±33 mm, ±33 mm) relative to the array center, a square of 66 mm side (93 mm diagonal; max inter-channel delay ≈ 4.3 samples at 16 kHz), from the official XVF3800 AEC_MIC_ARRAY_GEO. The x-axis points to the wearer's right, y-axis forward. WAV channel mapping (0-based): ch2 → rear-right, ch3 → rear-left, ch4 → front-left, ch5 → front-right. Azimuth is measured clockwise from the front (front = 0°, right = 90°, …, front-left = 315°).

Hint-generation window. The DOA hint is computed over the full 60 s clip (trailing segments < 3 s discarded) with the frame-level VAD and histogram-mode aggregation above. The direction recordings are controlled: each session contains a single source at a fixed, known azimuth (recorded per session in the metadata), so the whole-clip mode is a stable estimate and no event-onset alignment is needed.

We thank the reviewer again for the careful reading and constructive comments, which have genuinely helped us improve the accuracy of the paper's presentation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.