acceptodds
Under review as a conference paper at ICLR 2027

Retrieve-then-Reason: Always-On Proactive Assistance with On-Demand Reasoning

Abstract

Always-on proactive AI assistants must continuously handle streaming multimodal information, but invoking a multimodal large language model (MLLM) at every timestep is prohibitively expensive for on-device deployment. To address this challenge, we reformulate continuous proactive monitoring as a lightweight semantic retrieval problem and present PRISM (Proactive Reasoning via Semantic Monitoring), a cascaded framework that separates continuous retrieval from on-demand contextual reasoning. PRISM first identifies potentially relevant events with high recall and then filters them according to their contextual actionability, reducing false alarms and unnecessary MLLM invocations. Specifically, lightweight Visual and State Retrievers continuously retrieve instruction-relevant visual evidence and semantic states from the incoming multimodal stream leaving contextual actionability assessment to the subsequent verification stage. On-demand Verifier then evaluates whether each retrieved candidate satisfies the requested condition using the current multimodal context, recent history, and relevant semantic states before forwarding it to the MLLM-based Responder. On the PhoStream benchmark, PRISM improves performance on future-dependent proactive tasks, substantially reduces premature responses, and achieves a performance–energy Pareto improvement over direct MLLM invocation at every step. Our results show that lightweight semantic monitoring combined with contextual verification enables efficient continuous proactive assistance while reserving expensive reasoning for potentially actionable events.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Proactive agents, Always-on, On-device AI

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_n2M4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The core problem addressed in this paper is that resident AI assistants need to continuously monitor videos and conversation streams, but run large models at every step, and their battery capacity cannot withstand it. The author proposes the PRISM framework with the idea of doing cheap work all the time and doing expensive work as needed. Specifically, it is a three-stage pipeline: the first stage uses a lightweight encoder (22M parameter) to continuously search for "possibly relevant" candidates; The second level uses a validator of less than 1B, combined with context, to determine whether these candidates are "really worthy of action" and block false positives caused by "relevant but not yet timely"; It is only at the third level that the 4B model generates answers. In the experiment, PhoStream scored 40.0 overall, surpassing the best baseline. The most difficult Forward task increased from 7.7 to 22.7, premature response decreased from 81% to 9.4%, and energy consumption was only one-third of the best baseline.

Critical strengths
  1. The real pain point of edge deployment is that resident monitoring cannot run large models every step. The disassembly of retrieve verify respond separates expensive semantic judgments from continuous perception, with clear and reusable ideas, not just stack tricks.
  2. The Forward task increased from 7.7 to 22.7, premature response decreased from 81% to 9.4%, energy consumption decreased to one-third of the optimal baseline, and the main result, along with complete ablation and sensitivity analysis, provided a solid evidence chain.
Critical weaknesses
  1. The core selling point "Pareto improvement in performance energy consumption" is based on the assumption of fixed NPU power consumption and the calculation of model agents (using Qwen3 to impersonate Qwen3.5), and only calculates the prefill, which is purely a relative estimation rather than actual measurement. For a systematic paper that focuses on efficiency, this weakens the most weighty conclusion.
  2. The author admits that removing the verifier and allowing the 4B responder to make the final judgment directly increases the score by about 1.15 times. The high dependence of the benefits of the three-level cascade on conditions such as intervention sparsity is not universally valid, and the robustness of the method design is compromised.
Questions for authors

I have a few questions to confirm: Firstly, the energy consumption in Table 1 is based on the assumption of Qwen3 replacing Qwen3.5, fixed 1.75W power consumption, and only pre fill calculation. Have you ever verified it even on a small scale on real devices? If the agency relationship is not established, can the selling point of "3x power saving" still hold up? Secondly, you mentioned that after removing the verifier, the score actually increased by 1.15 times. So when is the three-level cascade really necessary? Have you ever considered having the 4B responder serve as both a gate control and a verifier? This directly affects my assessment of the robustness of the method. Thirdly, the visual vocabulary consists of COCO and Places365, with only about 600 semantic states. If the user's instructions exceed the scope of the vocabulary or if the language scene is changed, to what extent will the system degrade? Fourthly, SIR Bench only has 22 videos and 201 questions, and the dialogues and questions are still synthesized and filtered by Claude himself. With such a small scale, how strong conclusions do you think it can support?

6 comments
AuthorsOpenReview ↗

Rebuttal from Authors (1/3)

We thank Reviewer n2M4 for the careful review and the concrete questions. We deployed the actual Qwen3.5 models and the baselines on a Samsung Galaxy S26 Ultra and measured their energy on the device, and we evaluated the 4B Responder as its own gate and verifier on a continuous timeline.

W1. The core selling point "Pareto improvement in performance energy consumption" is based on the assumption of fixed NPU power consumption and the calculation of model agents (using Qwen3 to impersonate Qwen3.5), and only calculates the prefill, which is purely a relative estimation rather than actual measurement. For a systematic paper that focuses on efficiency, this weakens the most weighty conclusion.

Q1. I have a few questions to confirm: Firstly, the energy consumption in Table 1 is based on the assumption of Qwen3 replacing Qwen3.5, fixed 1.75W power consumption, and only pre fill calculation. Have you ever verified it even on a small scale on real devices? If the agency relationship is not established, can the selling point of "3x power saving" still hold up?

We have now measured the Table 1 models on a real device, not only on a small scale, and the 3× claim holds and widens: PRISM uses 0.26× the energy of the best baseline, Qwen3-VL-8B (submission estimate: 0.32×). Qwen3.5 could not yet be deployed on the device at submission time because of its new architecture; it now can. We measured the energy of the models on a Galaxy S26 Ultra and recomputed Table 1 (Table R3.1).

methodOverallsubmissionon-device
Direct Qwen3.5-0.8B23.81.001.00
Direct Qwen3.5-2B27.71.581.64
Direct Qwen3.5-4B30.02.243.52
Direct Qwen3.5-9B34.63.365.00
Direct Qwen3-VL-8B36.33.165.25
Direct Qwen2.5-Omni-7B29.12.964.87
Decoder-gated 0.8B → 4B30.93.234.51
Decoder-gated 0.8B → 9B35.44.355.98
PRISM40.01.001.37

Table R3.1. Normalized energy on PhoStream (direct Qwen3.5-0.8B = 1.00), submission estimate vs. on-device measurement. The revised Table 1 reports the same values.

  • The 3× claim holds and widens. On the device, larger models cost much more energy per call than the submission estimated, so the direct baselines rise sharply. PRISM runs the 0.8B Verifier only on retrieved candidates and calls the 4B Responder just 1.86 times per question. Even with its decode now charged, its energy advantage over the best baseline, Qwen3-VL-8B, widens from 3.16× to 3.83×.
  • The Pareto improvement holds. PRISM achieves the highest score (40.0) while using less energy than every baseline except direct Qwen3.5-0.8B. It scores 4.6 points higher than the decoder-gated two-stage baseline (0.8B → 9B) while using 4.36× less energy.
  • PRISM is charged its full cost. PRISM is charged the prefill and decode of every Verifier and Responder call, whereas each direct baseline is charged prefill plus a single output token per tick (e.g., a silent token).

Table R3.2 summarizes how each assumption of the submission is replaced.

assumptions in submissionupdated on-device measurement
fixed 1.75 W powerpower measured per model and per phase; decode draws 0.44–0.66× the prefill power (Table R3.3)
Qwen3 as a proxy for Qwen3.5Qwen3.5 and the baselines measured directly; the Qwen3 proxies are also close (same-device energy ratio Qwen3-4B / Qwen3.5-4B: 0.94–1.09, Table R3.4)
prefill onlydecode included: 67% of a Verifier call's energy and 23% of a Responder call's energy

Table R3.2. Each assumption of the submission's Appendix A.1 and its on-device replacement.

AuthorsOpenReview ↗

Rebuttal from Authors (2/3)

We give the details of each item below.

ModelPrefill powerDecode powerPrefill energyEnergy per decoded tokenEnergy per call
Qwen3-0.6B1.000.660.921.391.27
Qwen3.5-0.8B1.000.591.001.001.00
Qwen3.5-2B1.230.561.781.781.78
Qwen3-4B1.080.483.653.413.47
Qwen3.5-4B1.140.503.843.383.50
Qwen2.5-Omni-7B1.090.595.155.655.52
Qwen3-8B1.070.565.736.095.99
Qwen3.5-9B1.050.585.656.336.15

Table R3.3. Power and energy per model and per phase, normalized to Qwen3.5-0.8B (= 1.00), at 1,686 input tokens (the mean Verifier input) and 120 output tokens.

  • Power is measured per model and per phase. As Table R3.3 shows, decode draws much less power than prefill for every model, only 0.44–0.66× of its prefill power.
  • Decode draws less power, but its latency drives up the energy. Decoding takes much longer than prefill, so despite its lower power it accounts for 67% of a Verifier call's energy. It is now charged in full.
Proxy pairOutput tokens1,024 in1,686 in2,048 in4,096 in
Qwen3-0.6B / Qwen3.5-0.8B10.950.920.971.15
Qwen3-0.6B / Qwen3.5-0.8B1201.001.271.071.38
Qwen3-4B / Qwen3.5-4B10.990.951.021.09
Qwen3-4B / Qwen3.5-4B1200.940.990.991.08

Table R3.4. Per-call energy of each Qwen3 proxy relative to the corresponding Qwen3.5 model, measured on the same device (1.00 = identical).

  • Qwen3 consumes nearly the same energy as Qwen3.5. As Table R3.4 shows, the ratios are close to 1.00, so the proxies in the submission did not deviate much from the actual deployment.
  • For long inputs and outputs, Qwen3 consumes more energy. Owing to the linear attention of Gated DeltaNet, Qwen3.5-0.8B becomes cheaper than Qwen3-0.6B as sequences grow longer. The proxy therefore did not favor PRISM; if anything, it overestimated the Verifier's cost.

We have revised Appendix A.1 (rewritten, with the new Fig. 7), Table 1, Table 2, Sec. 4.2, the introduction's efficiency statement, and the energy figures (Fig. 1(c), Fig. 5(a–b), and Fig. 8) accordingly (in blue).

W2. The author admits that removing the verifier and allowing the 4B responder to make the final judgment directly increases the score by about 1.15 times. The high dependence of the benefits of the three-level cascade on conditions such as intervention sparsity is not universally valid, and the robustness of the method design is compromised.

Q2. Secondly, you mentioned that after removing the verifier, the score actually increased by 1.15 times. So when is the three-level cascade really necessary? Have you ever considered having the 4B responder serve as both a gate control and a verifier? This directly affects my assessment of the robustness of the method.

The three-stage cascade is necessary whenever the assistant monitors continuously, because it is what makes always-on monitoring affordable. Removing the Verifier, or letting the 4B Responder serve as both the gate control and the verifier, does raise the PhoStream score: the 4B Responder then makes the final decision on candidates the Verifier would have rejected, so fewer valid interventions are falsely rejected. This gain, however, requires running the 4B model on far more ticks: under the extended monitoring of Table 2, these configurations consume 2.0× and 5.5× the energy of PRISM.

configurationPhoStream score (× PRISM)4B Responder calls (% of ticks)energy (× PRISM)
PRISM (Retriever → Verifier → Responder)1.0010.2%1.0
w/o Verifier (Retriever → Responder)1.1536.9%2.0
4B Responder as both gate control and verifier1.26100%5.5

Table R3.5. Score and cost of the three configurations. PhoStream score is the Overall score relative to PRISM, following the official protocol on the full benchmark; Responder call rates and energy are measured under the extended monitoring of Table 2 (25 videos, 2.6 hours of stream), with energy from the on-device measurements.

  • The score gain comes from fewer false rejections. Without the Verifier, the 4B Responder also judges the candidates the Verifier would have rejected and recovers some valid interventions among them, which raises the score by about 1.15×.
  • A 4B gate makes always-on monitoring prohibitively expensive. As Table R3.5 shows, the Verifier forwards only 10.2% of ticks to the 4B Responder, compared with 36.9% without it and 100% when the 4B Responder serves as its own gate. The energy grows accordingly, to 2.0× and 5.5× that of PRISM, far beyond the score gain of at most 1.26×, and the extra cost accumulates with every hour of monitoring, which an always-on assistant on a battery-powered device cannot sustain.
AuthorsOpenReview ↗

Rebuttal from Authors (3/3)

Q3. Thirdly, the visual vocabulary consists of COCO and Places365, with only about 600 semantic states. If the user's instructions exceed the scope of the vocabulary or if the language scene is changed, to what extent will the system degrade?

We thank the reviewer for this interesting question. PRISM does not require the requested concept to be in the vocabulary. The Retriever wakes the Verifier by embedding similarity, so an instruction about a missing concept enrolls its nearest concepts instead, and frames showing the requested concept still match them.

To test this, we removed the requested concept from the vocabulary and checked whether its images still wake the Verifier. Table R3.6 shows examples, and Table R3.7 repeats this for every COCO and Places365 concept.

removed conceptconcepts enrolled insteadwake recall: in vocabulary → removed
elephanthorse, cow, bear, giraffe, sheep98% → 90%
pizzapizzeria, sandwich, fast food restaurant, restaurant, delicatessen95% → 93%
bookstoreindoor library, gift shop, outdoor library, drugstore, book97% → 93%

Table R3.6. Examples with the requested concept removed from the vocabulary (e.g., "Tell me when you see an elephant"). Wake recall is the share of images containing the concept that wake the Verifier.

requested conceptCOCO wake recallfalse-wake ratePlaces365 wake recallfalse-wake rate
in vocabulary68.0%6.8%84.2%4.8%
removed56.8% (0.84×)7.7%70.6% (0.84×)5.3%
removed with its supercategory51.1% (0.75×)7.0%––

Table R3.7. Wake recall on images containing the requested concept and false-wake rate on images without it, averaged over all 80 COCO and 365 Places365 concepts and two instruction templates.

  • The system degrades gracefully. As Tables R3.6 and R3.7 show, without the requested concept PRISM keeps 0.84× of its wake recall on both datasets (0.75× even with all related concepts removed), while false wakes change by less than one percentage point.

A missing concept can also be added by precomputing a few text embeddings, with no retraining. On the language side, the State Retriever is never trained on PhoStream, so all PhoStream results already come from conversations outside its training distribution.

We have added this analysis to Sec. 4.2 (new Table 3, in blue).

Q4. Fourthly, SIR Bench only has 22 videos and 201 questions, and the dialogues and questions are still synthesized and filtered by Claude himself. With such a small scale, how strong conclusions do you think it can support?

We thank the reviewer for this question. SIR-Bench supports a focused conclusion: PRISM's Retriever and Verifier make specific, relevant decisions about an implicit need for help, which raise accuracy from 60.7% to 65.2% with the same Responder (Table 5). No intervention-timing or energy claim in the paper relies on it.

  • Real requests may reflect implicit needs. PhoStream specifies the target through explicit cues such as "Wait until …," which users rarely say in daily life. A user is more likely to mention feeling tired and unwell, and a helpful assistant should then point to a nearby place to sit with water (Fig. 6). SIR-Bench is built so that a model cannot answer without inferring this unstated need.
  • It tests whether the Retriever and Verifier make specific decisions. A useful decision is not a generic "the user needs help" but a concrete state, such as physical discomfort or the user's current surroundings, that points to the right intervention. With the same 4B Responder, adding the retrieved user states and the Verifier summary raises accuracy by 4.5 points.

To verify this filtering beyond Claude, we gave model families that played no role in construction only the question and the four options (Table R3.8).

modelaccuracy without input
GPT-5.530.8%
GPT-5.4-mini21.4%
Gemini-3.5-Flash25.4%
Gemini-2.5-Flash30.3%
Qwen3.5-4B (submission)25.4%

Table R3.8. Accuracy when only the question and the four options are given (chance = 25%). Same prompt and option order as the submission.

  • SIR-Bench cannot be solved without the input. As Table R3.8 shows, every model stays close to chance (21–31%) without the input, whereas Qwen3.5-4B reaches 60.7% with the video and conversation.
  • The filtering holds beyond Claude. GPT, Gemini, and Qwen models all stay near chance, so the filtering is not specific to the generator.

We have clarified the intent of SIR-Bench in Sec. 4.3, and its construction together with the no-input check across model families in the Appendix (in blue). Inferring users' implicit needs from everyday context is an important yet underexplored challenge; we see SIR-Bench as a starting point for this direction and plan to expand it, guided by community feedback.

Reviewer_SaUVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PRISM, which decomposes continuous proactive assistance into three stages: lightweight semantic retrieval, on-demand context validation, and selective response generation. Visual retrieval matches user-relevant concepts, and state retrieval extracts active task, waiting and user context. A small Verifier judges whether candidate triggers satisfy intervention criteria, preventing large response models from activating on semantically relevant but inactionable inputs.

Critical strengths
  1. A clear intermediate criterion for intervention timing is introduced. The Verifier distinguishes between observing a relevant object and the satisfaction of user-specified triggering conditions. On PhoStream, the Forward metric rises from 7.7 for the strongest direct-response baseline to 22.7, while premature triggering drops from 81.3% to 9.4%, confirming the policy alters intervention behaviour.
  2. Comprehensive component and threshold analysis. The authors evaluate retrieval, verification, waiting thresholds, context windows and alternative routing rules, and explicitly disclose that removing the Verifier can raise overall scores. This enables readers to assess tradeoffs across the cascade rather than viewing only a single optimal operating point.
Critical weaknesses
  1. Core energy conclusions rest on an incomplete cost model with proxy errors. Appendix A.1 only accounts for prefill cost, assumes fixed power draw, uses external checkpoints as model proxies, and extrapolates for long inputs. The Verifier is assumed to generate ~120 output tokens, yet decoding cost is excluded from primary energy comparisons. ASR is processed offline and omitted from continuous runtime overhead. The authors acknowledge these scope limits, but these omissions may reverse relative advantages between routing configurations, especially three-stage versus two-stage pipelines.
  2. Evaluation does not validate performance under continuous monitoring. While premature interventions decrease, the Forward no-response rate rises from 5.7% to 48.0%. Improvements cannot be interpreted without ruling out excessive silence. The 25 extended long-horizon test samples only measure call frequency rather than response correctness. The sparse fixed-tick evaluation protocol is not cleanly aligned with continuous monitoring cost. SIR-Bench contains only 201 synthetic multiple-choice questions over 22 videos; it evaluates cue selection but not the core decision of whether or when to intervene.
Questions for authors
  1. After including Verifier and Responder decoding plus live upstream preprocessing overhead, does the three-stage pipeline retain favourable quality–latency–energy tradeoffs against two-stage baselines?
  2. How are quality evaluations at six discrete timestamps, the average 18.63 monitoring ticks and extended monitoring regime costed? Can quality and cost be reported jointly on a unified continuous timeline?
  3. At matched event recall or false intervention rates, does PRISM outperform a simple threshold-calibrated cascade? Please report false interventions per hour, event miss rates and response latency.
7 comments
AuthorsOpenReview ↗

Rebuttal from Authors (1/3)

We thank Reviewer SaUV for the precise review. We re-measured energy on a real device for PRISM and the baselines, and added a continuous-timeline evaluation that reports quality and cost jointly.

W1. Core energy conclusions rest on an incomplete cost model with proxy errors. Appendix A.1 only accounts for prefill cost, assumes fixed power draw, uses external checkpoints as model proxies, and extrapolates for long inputs. The Verifier is assumed to generate ~120 output tokens, yet decoding cost is excluded from primary energy comparisons. ASR is processed offline and omitted from continuous runtime overhead.

Q1. After including Verifier and Responder decoding plus live upstream preprocessing overhead, does the three-stage pipeline retain favourable quality–latency–energy tradeoffs against two-stage baselines?

Qwen3.5's new architecture could not yet be deployed on the device at submission time; it now can, so we measured the energy of the models on a Galaxy S26 Ultra (Table R2.1).

assumptions in submissionupdated on-device measurement
prefill onlydecode included: 67% of a Verifier call's and 23% of a Responder call's energy
fixed power drawmeasured per model and phase; decode draws 0.44–0.66× the prefill power (Table R2.2)
ASR omittedlive ASR charged per second of audio (3.6% of PRISM's total)
Qwen3 proxiesQwen3.5 and the baselines measured directly; the proxies are close (Qwen3-4B / Qwen3.5-4B energy: 0.94–1.09, Table R2.4)
interpolation / extrapolationmeasured energy is linear in input and output length (new Fig. 7)

Table R2.1. Appendix A.1 assumptions and their replacements.

ModelPrefill powerDecode powerPrefill energyEnergy per decoded tokenEnergy per call
Qwen3-0.6B1.000.660.921.391.27
Qwen3.5-0.8B1.000.591.001.001.00
Qwen3.5-2B1.230.561.781.781.78
Qwen3-4B1.080.483.653.413.47
Qwen3.5-4B1.140.503.843.383.50
Qwen2.5-Omni-7B1.090.595.155.655.52
Qwen3-8B1.070.565.736.095.99
Qwen3.5-9B1.050.585.656.336.15

Table R2.2. Per-model, per-phase power and energy, normalized to Qwen3.5-0.8B (= 1.00), at 1,686 input tokens (the mean Verifier input) and 120 output tokens.

  • Power is measured per model and per phase. As Table R2.2 shows, decode draws only 0.44–0.66× the prefill power for every model.
  • Decode draws less power, but its latency drives up the energy. Decoding takes much longer, so it accounts for 67% of a Verifier call's energy; it is now charged in full.
ComponentModelEnergy per callCalls per questionEnergy per questionShare of PRISM
Retriever (State + Visual)E5-small + PE-Core0.0614.140.064%
ASRQwen3-ASR-1.7B0.0514.140.054%
VerifierQwen3.5-0.8B0.929.980.6548%
ResponderQwen3.5-4B4.591.860.6044%
PRISM total1.37100%
Direct Qwen3.5-0.8BQwen3.5-0.8B1.0014.141.00
Direct Qwen3.5-4BQwen3.5-4B3.5214.143.52

Table R2.3. PRISM energy breakdown. Energy per call and per question are normalized to direct Qwen3.5-0.8B (= 1.00).

  • Every component is measured on the device. As Table R2.3 shows, the Verifier and Responder account for 92% of PRISM's energy.
  • ASR is not a bottleneck. ASR processes 32 s of audio at once (about 430 input tokens), so it is far cheaper than the other components; Table R2.3 reports its energy per second of audio (one call / 32). With little speech, its cost is even lower.
Proxy pairOutput tokens1,024 in1,686 in2,048 in4,096 in
Qwen3-0.6B / Qwen3.5-0.8B10.950.920.971.15
Qwen3-0.6B / Qwen3.5-0.8B1201.001.271.071.38
Qwen3-4B / Qwen3.5-4B10.990.951.021.09
Qwen3-4B / Qwen3.5-4B1200.940.990.991.08

Table R2.4. Per-call energy of each Qwen3 proxy relative to its Qwen3.5 model.

  • Qwen3 consumes nearly the same energy as Qwen3.5. As Table R2.4 shows, the ratios are close to 1.00, so the submission's proxies stayed close to the actual deployment.
  • For long inputs and outputs, Qwen3 consumes more energy. With Gated DeltaNet's linear attention, Qwen3.5-0.8B becomes cheaper than Qwen3-0.6B on longer sequences, so the proxy did not favor PRISM; if anything, it overestimated the Verifier's cost.

On interpolation and extrapolation, the new Fig. 7 shows that the measured per-call energy of every model is close to linear in input and output length: a linear fit explains at least 94% of the variance in every slice of the measured grid, and at least 99% in 92% of them.

AuthorsOpenReview ↗

Rebuttal from Authors (2/3)

Based on this energy model, we recomputed Table 1.

methodOverallsubmissionon-device
Direct Qwen3.5-0.8B23.81.001.00
Direct Qwen3.5-2B27.71.581.64
Direct Qwen3.5-4B30.02.243.52
Direct Qwen3.5-9B34.63.365.00
Direct Qwen3-VL-8B36.33.165.25
Direct Qwen2.5-Omni-7B29.12.964.87
Decoder-gated 0.8B → 4B30.93.234.51
Decoder-gated 0.8B → 9B35.44.355.98
PRISM40.01.001.37

Table R2.5. Normalized energy on PhoStream (direct Qwen3.5-0.8B = 1.00), submission estimate vs. on-device measurement. Revised Table 1 reports the same values.

Table R2.5 recomputes Table 1 with the new on-device measurements.

  • The 3× claim holds and widens. On the device, larger models cost much more energy per call than the submission estimated, so the direct baselines rise sharply. PRISM runs the 0.8B Verifier on the retrieved candidates and calls the 4B Responder only 1.86 times per question, so even with its own decode, ASR, and retrievers now charged, its energy advantage over the best baseline, Qwen3-VL-8B, widens from 3.16× to 3.83×.
  • PRISM outperforms the two-stage baseline. Even in this setting, PRISM scores 4.6 points higher than the decoder-gated two-stage baseline (0.8B → 9B) while using 4.36× less energy.

We have revised Appendix A.1 (rewritten, with the new Fig. 7), Table 1, Table 2, Sec. 4.2, and the energy figures (Fig. 1(c), Fig. 5(a–b), and Fig. 8) accordingly (in blue).

W2-1. Evaluation does not validate performance under continuous monitoring. While premature interventions decrease, the Forward no-response rate rises from 5.7% to 48.0%. Improvements cannot be interpreted without ruling out excessive silence. The 25 extended long-horizon test samples only measure call frequency rather than response correctness. The sparse fixed-tick evaluation protocol is not cleanly aligned with continuous monitoring cost.

Q2. How are quality evaluations at six discrete timestamps, the average 18.63 monitoring ticks and extended monitoring regime costed? Can quality and cost be reported jointly on a unified continuous timeline?

We thank the reviewer for raising this point, and we show below that PRISM's Forward silence is not excessive. PRISM is never told whether a question is Instant, Backward, or Forward; it has to infer from the stream whether the requested evidence is already present or still to come. Its high Forward score is therefore not obtained by simply staying silent. Regarding the sparse fixed-tick protocol, Table 1 follows the official evaluation time points and scoring, so PRISM is scored exactly like the published baselines. PRISM itself runs continuously: the qualitative results in Fig. 4 and Fig. 11 come from runs that process every 1-s tick of the stream. Below, we further detail how PRISM behaves on a continuous timeline.

τ_pendInstant NRBackward NRForward NR
0.0558.8%62.7%51.2%
0.2525.1%26.7%49.9%
0.4514.6%13.9%49.6%
0.85 (default)6.9%6.2%48.0%

Table R2.6. No-response rate (NR) per task on the full benchmark across τ_pend.

  • Silence is not a prior. As Table R2.6 shows, PRISM can also choose silence on Instant and Backward questions. If we tune it to pend more often (small τ_pend), it loses substantial performance on these tasks.
  • Silence is the model's decision. Raising τ_pend from 0.05 to our default of 0.85 lowers Forward NR by only about 3 points, whereas Instant and Backward NR drop by more than 50 points. PRISM thus decides on its own when to stay silent; its performance does not come from simply staying silent.
false interventions / hrresponses in the answer window4B calls / qenergy (× PRISM)
PRISM15866.0%4.021.00
w/o retrieval gating19363.9%4.451.20
w/o Verifier (2-stage)30662.9%8.801.31

Table R2.7. Quality and cost on the unified continuous timeline.

Unified continuous timeline. Next, we show how often PRISM intervenes falsely and how much energy it consumes in a continuous setting, together with ablations of each component. On the 60-question Forward sample, every configuration is run at every 1-s tick from the query until 6 s after the gold time. A response before the gold time is a false intervention; a response within [t_gold, t_gold + 6 s] falls in the answer window.

  • PRISM makes the fewest false interventions in the continuous setting. Through the Verifier and retrieval gating, PRISM makes half as many false interventions as the two-stage pipeline. It also has the largest share of its responses inside the answer window, and it uses the least energy.

We have added this continuous-timeline evaluation to Appendix A.7 (in blue).

AuthorsOpenReview ↗

Rebuttal from Authors (3/3)

W2-2. SIR-Bench contains only 201 synthetic multiple-choice questions over 22 videos; it evaluates cue selection but not the core decision of whether or when to intervene.

SIR-Bench complements PhoStream rather than replacing it: whether and when to intervene is evaluated on PhoStream (Tables R2.6, R2.7, and R2.9), while SIR-Bench evaluates what the assistant must infer when the user never asks for help.

  • Real requests may reflect implicit needs. PhoStream specifies the target through explicit cues such as "Wait until …," which users rarely say in daily life. A user is more likely to mention feeling tired and unwell, and a helpful assistant should then point to a nearby place to sit with water (Fig. 6). SIR-Bench is built so that a model cannot answer without inferring this unstated need.
  • It tests whether the Retriever and Verifier make specific decisions. A useful decision is a concrete state, such as physical discomfort or the user's surroundings, rather than a generic "the user needs help." With the same 4B Responder, adding these states and the Verifier summary raises accuracy from 60.7% to 65.2% (Table 5).

To check that SIR-Bench requires the input, we gave models not used in construction only the question and the four options (Table R2.8).

modelaccuracy without input
GPT-5.530.8%
GPT-5.4-mini21.4%
Gemini-3.5-Flash25.4%
Gemini-2.5-Flash30.3%
Qwen3.5-4B (submission)25.4%

Table R2.8. SIR-Bench accuracy with only the question and options (chance = 25%), using the submission's prompt and option order.

  • SIR-Bench cannot be solved without the input. As Table R2.8 shows, every model stays close to chance (21–31%) without the input, whereas Qwen3.5-4B reaches 60.7% with the video and conversation.
  • The filtering holds beyond Claude. All three model families stay near chance, so the filtering is not specific to the generator.

We have clarified the intent of SIR-Bench in Sec. 4.3, and its construction and no-input check in the Appendix (in blue). Inferring users' implicit needs from everyday context is an important yet underexplored challenge; we see SIR-Bench as a starting point for this direction and plan to expand it, guided by community feedback.

Q3. At matched event recall or false intervention rates, does PRISM outperform a simple threshold-calibrated cascade? Please report false interventions per hour, event miss rates and response latency.

PRISM outperforms a threshold-calibrated cascade at every operating point of the sweep: at a matched false-intervention rate, it places twice as many of its responses at the right time. We use the decoder-gated cascade of Table 1 (Qwen3.5-0.8B gate → Qwen3.5-4B Responder), which invokes the Responder when the gate's per-tick probability of "Yes" (answerable now) exceeds a threshold. We swept this threshold around its default of 0.5 on the continuous timeline (60 Forward questions, every 1-s tick from the query until 6 s after the gold time) to match PRISM's false-intervention rate.

methodgate thresholdgate opens (% of ticks)false interventions / hrresponses in the answer windowenergy (× PRISM)
Decoder-gated two-stage0.5 (default)30.2%44834.1%1.19
Decoder-gated two-stage0.612.8%18033.3%0.84
Decoder-gated two-stage0.73.0%3237.5%0.64
PRISM––15866.0%1.00

Table R2.9. Decoder-gated two-stage cascade with its Yes/No threshold swept, compared with PRISM on the continuous timeline. A response before the gold time is a false intervention; a response within [t_gold, t_gold + 6 s] falls in the answer window. Energy follows the Table 1 accounting with on-device per-call costs.

  • At a matched false-intervention rate, PRISM places twice as many of its responses at the right time. At a threshold of 0.6, the cascade makes 180 false interventions per hour, comparable to PRISM's 158, yet only 33.3% of its responses fall in the answer window, versus 66.0% for PRISM.
  • The cascade's lower cost comes from indiscriminate rejection. Its gate opens on only 12.8% of ticks at a threshold of 0.6 and on 3.0% at 0.7, yet its responses are no better timed (33–38% in the answer window). It reduces false interventions by declining to respond, not by recognizing when the requested event has occurred, so these operating points are impractical for an assistant.
  • No threshold lets the gate separate actionable from premature moments. At its default threshold, the cascade uses 1.19× PRISM's energy yet makes 2.8× as many false interventions (448 vs. 158 per hour). Across the sweep, its answer-window share stays at 33–38%, versus two thirds for PRISM.

We have added this threshold sweep to Sec. 4.2 (new Table 4, in blue).

Reviewer_v2uMOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PRISM, aiming for more efficient proactive assistance by using lightweight retrieval to monitor visual/textual inputs and selectively calling VLMs to verify events and generate responses. They test the method with Qwen models on PhoStream and their proposed SIR-Bench. Results show that PRISM achieves the highest overall score on PhoStream, using around one-third of the estimated inference energy of the best-performing baseline.

Critical strengths
  1. Experiments seem solid and support the paper’s claim.
Critical weaknesses
  1. It’s unclear how many useful interventions are lost during filtering. One can track which stage rejects each valid intervention, and compare performance and cost w or w/o each stage..

  2. Tab8 shows forward score drops from 22.2 to 6.4 when removing suppression caching. How much improvement comes from retrieval and verification versus the suppression rule?.

  3. The energy saving is estimated by computing prefill costs using proxy models, but decoding and ASR preprocessing are not included. Is there a way to measure the costs of the two missing parts to have a better estimate of the whole system’s cost?

Questions for authors
  1. Fig2 contains too many details that might distract readers.
5 comments
AuthorsOpenReview ↗

Rebuttal from Authors (1/3)

We thank Reviewer v2uM for the constructive comments. We ran the requested stage-wise analyses, and we deployed the actual Qwen3.5 models and the baselines on a Samsung Galaxy S26 Ultra to measure the whole system's energy, including decode and ASR.

W1. It’s unclear how many useful interventions are lost during filtering. One can track which stage rejects each valid intervention, and compare performance and cost w or w/o each stage.

We thank the reviewer for this helpful suggestion. When the evidence is already in the stream, the filters lose almost no useful interventions; on Forward questions, PRISM delivers 3.3× more timely interventions than the strongest baseline. We traced every question of the full PhoStream benchmark (5,572 questions) to the stage that handled it.

outcomeInstant (1,647)Backward (1,123)Forward (2,802)
rejected by the Retriever0.2%–20.0%
rejected by the Verifier0.1%0.1%24.1%
Responder chose to stay silent6.5%6.1%3.7%
answered in the scoring window92.8%93.6%42.6%
answered outside the window (late / early)0.4%0.2%9.4%

Table R1.1. Stage at which each valid intervention was handled, as a share of all questions of each type (full benchmark, 5,572 questions). The pending-state hold is counted with the Retriever.

  • On Instant and Backward questions, the filters almost never block a valid intervention. When the evidence is already in the stream, the Retriever and Verifier together reject only 0.3% (Instant) and 0.1% (Backward) of the questions.
  • On Forward questions, the filters create useful interventions rather than lose them. In these questions, the requested event has not yet happened when the question is asked, so the Retriever and Verifier must wait until it occurs. Without such filters, direct models answer too early: the strongest direct baseline, Qwen3-VL-8B, answers only 12.9% of Forward questions at the right time (PC, Table 1). PRISM answers 42.6%, 3.3× more.

To compare cost with and without each stage, we use the extended monitoring setting of Table 2 (25 videos, 2.6 hours of stream), where the assistant keeps monitoring long after each request.

configuration4B Responder calls (% of ticks)energy (× PRISM)
PRISM10.2%1.0
w/o Verifier36.9%2.0
w/o both (4B Responder at every tick)100%5.5

Table R1.2. PRISM with and without each stage under extended monitoring. Energy is from the on-device measurements (W3).

  • The filters keep the 4B Responder idle most of the time. The Verifier forwards only 10.2% of ticks to the 4B Responder. Without it, 36.9% of ticks reach the 4B Responder (2.0× the energy); without both filters, every tick does (5.5×).

Together, these results show that PRISM's early stages defer judgment on future-dependent cues rather than intervening prematurely, letting this evidence mature into timely interventions while invoking the costly 4B Responder on only 10.2% of ticks.

W2. Tab8 shows forward score drops from 22.2 to 6.4 when removing suppression caching. How much improvement comes from retrieval and verification versus the suppression rule?

We thank the reviewer for this insightful question. Two results answer it: (1) retrieval and verification alone already answer too early less often than any baseline, and (2) suppression is meaningful only on top of retrieval and verification. Suppression keeps a "not yet" decision for 5 s: if a method stays silent right after a question, its responses in the next 5 s are suppressed. To separate the two, we applied the same suppression to every baseline in Table 1.

methodanswered too early (no suppression)answered too early (same 5 s suppression)
Qwen3.5-4B99.4%97.1%
Qwen3.5-9B86.4%77.8%
Qwen3-VL-8B81.3%76.7%
Qwen2.5-Omni-7B97.9%94.1%
Decoder-gated 0.8B → 9B89.3%86.6%
PRISM73.1%9.4%

Table R1.3. Forward questions (2,802) with and without the same 5 s suppression. Answering too early means responding before the requested event happens.

  • (1) Retrieval and verification alone already beat every baseline. Without suppression, PRISM answers too early on 73.1% of Forward questions, below every baseline (81.3–99.4%).
  • (2) Suppression is meaningful only with retrieval and verification. Applied to the baselines, the same suppression lowers too-early answers by only 2–9 points; on PRISM, it lowers them by 64 points (73.1% → 9.4%). Suppression only keeps a "not yet" decision, so it works for PRISM because PRISM knows when not to answer.
  • The suppression length needs no fine tuning. Suppression only needs to keep the decision right after the question: any length of 2–5 s gives a similar Forward score (22.2–24.2).
AuthorsOpenReview ↗

Rebuttal from Authors (2/3)

W3. The energy saving is estimated by computing prefill costs using proxy models, but decoding and ASR preprocessing are not included. Is there a way to measure the costs of the two missing parts to have a better estimate of the whole system’s cost?

We thank the reviewer for this suggestion. Both costs are now measured on a real device. Qwen3.5 could not yet be deployed on the device at submission time because of its new architecture; it now can. We therefore measured the energy of the models, including decode and ASR, on a Galaxy S26 Ultra. Table R1.4 summarizes what changed.

assumptions in submissionupdated on-device measurement
prefill onlydecode included: 67% of a Verifier call's energy and 23% of a Responder call's energy
ASR omittedlive ASR charged for every second of audio (3.6% of PRISM's total)
Qwen3 proxiesQwen3.5 and the baselines measured directly; the Qwen3 proxies are also close (same-device energy ratio Qwen3-4B / Qwen3.5-4B: 0.94–1.09, Table R1.7)
fixed power drawpower measured per model and per phase; decode draws 0.44–0.66× the prefill power (Table R1.5)
interpolation / extrapolationmeasured energy is linear in input and output length (Fig. 7 in the updated manuscript)

Table R1.4. Each assumption of the submission's energy estimate and its on-device replacement.

ModelPrefill powerDecode powerPrefill energyEnergy per decoded tokenEnergy per call
Qwen3-0.6B1.000.660.921.391.27
Qwen3.5-0.8B1.000.591.001.001.00
Qwen3.5-2B1.230.561.781.781.78
Qwen3-4B1.080.483.653.413.47
Qwen3.5-4B1.140.503.843.383.50
Qwen2.5-Omni-7B1.090.595.155.655.52
Qwen3-8B1.070.565.736.095.99
Qwen3.5-9B1.050.585.656.336.15

Table R1.5. Per-model, per-phase power and energy, normalized to Qwen3.5-0.8B (= 1.00), at 1,686 input tokens (the mean Verifier input) and 120 output tokens.

  • Power is measured per model and per phase. As Table R1.5 shows, decode draws much less power than prefill for every model, only 0.44–0.66× of its prefill power.
  • Decode draws less power, but its latency drives up the energy. Decoding takes much longer than prefill, so despite its lower power it accounts for 67% of a Verifier call's energy.
ComponentModelEnergy per callCalls per questionEnergy per questionShare of PRISM
Retriever (State + Visual)E5-small + PE-Core0.0614.140.064%
ASRQwen3-ASR-1.7B0.0514.140.054%
VerifierQwen3.5-0.8B0.929.980.6548%
ResponderQwen3.5-4B4.591.860.6044%
PRISM total1.37100%
Direct Qwen3.5-0.8B BaselineQwen3.5-0.8B1.0014.141.00
Direct Qwen3.5-4B BaselineQwen3.5-4B3.5214.143.52

Table R1.6. Whole-system energy breakdown of PRISM, including ASR and the retrievers. Energy per call is normalized to one direct Qwen3.5-0.8B call, and energy per question to direct Qwen3.5-0.8B (= 1.00).

  • The whole system is measured. As Table R1.6 shows, PRISM's energy is broken down from on-device measurements of prefill, decode, ASR, and both retrievers. Most of it goes to the Verifier and Responder.
  • ASR is a small cost. ASR does not run at every timestep: it transcribes 32 s of audio per call from an input of only about 430 tokens, which makes it far cheaper than the other components. Table R1.6 lists its energy per second of audio (one call divided by 32). With little speech in the scene, the cost is even lower.
Proxy pairOutput tokens1,024 in1,686 in2,048 in4,096 in
Qwen3-0.6B / Qwen3.5-0.8B10.950.920.971.15
Qwen3-0.6B / Qwen3.5-0.8B1201.001.271.071.38
Qwen3-4B / Qwen3.5-4B10.990.951.021.09
Qwen3-4B / Qwen3.5-4B1200.940.990.991.08

Table R1.7. Per-call energy of each Qwen3 proxy relative to the corresponding Qwen3.5 model, measured on the same device (1.00 = identical).

  • The proxies were close to the actual models. As Table R1.7 shows, the ratios are near 1.00, so the proxies in the submission did not deviate much from the actual deployment.
  • The proxies did not favor PRISM. For long inputs and outputs, Qwen3-0.6B costs more than Qwen3.5-0.8B, which benefits from the linear attention of Gated DeltaNet. If anything, the proxy overestimated the Verifier's cost.

Measured energy is also close to linear in both input and output length (new Fig. 7): a linear fit explains at least 94% of the variance in every slice of the measured grid, and at least 99% in 92% of them, so the interpolation in Appendix A.1 introduces little error.

AuthorsOpenReview ↗

Rebuttal from Authors (3/3)

With these measurements, we recomputed Table 1.

methodOverallsubmissionon-device
Direct Qwen3.5-0.8B23.81.001.00
Direct Qwen3.5-2B27.71.581.64
Direct Qwen3.5-4B30.02.243.52
Direct Qwen3.5-9B34.63.365.00
Direct Qwen3-VL-8B36.33.165.25
Direct Qwen2.5-Omni-7B29.12.964.87
Decoder-gated 0.8B → 4B30.93.234.51
Decoder-gated 0.8B → 9B35.44.355.98
PRISM40.01.001.37

Table R1.8. Normalized energy on PhoStream (direct Qwen3.5-0.8B = 1.00), submission estimate vs. on-device measurement. The revised Table 1 reports the same values.

  • The energy saving holds and widens with the full system cost. On the device, larger models cost much more energy per call than the submission estimated, so the direct baselines rise sharply. PRISM runs the 0.8B Verifier only on retrieved candidates and calls the 4B Responder just 1.86 times per question. Even with its decode, ASR, and retrievers now charged, its energy advantage over the best baseline, Qwen3-VL-8B, widens from 3.16× to 3.83×.
  • PRISM also outperforms the two-stage baseline. PRISM scores 4.6 points higher than the decoder-gated two-stage baseline (0.8B → 9B) while using 4.36× less energy.

We have revised Appendix A.1 (rewritten, with the new Fig. 7), Table 1, Table 2, Sec. 4.2, and the energy figures (Fig. 1(c), Fig. 5(a–b), and Fig. 8) accordingly (in blue).

Q1. Fig2 contains too many details that might distract readers.

Thank you for this suggestion. We have updated the manuscript accordingly: Fig. 2 is now split into an overview of the Retriever → Verifier → Responder cascade (Fig. 2) and the component details (Fig. 3), so the overview shows only the main flow (in blue).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response from Authors

We thank all reviewers for their constructive feedback. All three reviewers raised the energy analysis, and we have addressed it directly: we deployed the actual Qwen3.5 models and the baselines on a Samsung Galaxy S26 Ultra and measured their per-call energy on the device.

What we measured. At submission time, Qwen3.5's new architecture (Gated DeltaNet) could not yet be deployed on the device, so the submission estimated energy from NPU latency profiles of Qwen3 checkpoints at a fixed power. The deployment is now possible, and we have deployed the Qwen models on the device.

  • Deployed models. Qwen3.5-0.8B/2B/4B/9B, Qwen3-VL-8B, and Qwen2.5-Omni-7B.
  • Unified protocol for all models. All models run on the device NPU and are measured with a unified protocol: whole-device battery power above the idle baseline, sampled every 100 ms, with the battery cooled to 30 °C before each model.
  • Decoding included. Each model is profiled over a grid of input lengths (1,024–4,096 tokens) and output lengths (1–120 tokens).
  • Full benchmark. Call counts are profiled over all 5,572 PhoStream questions (14.14 ticks per question on average), the same questions on which Table 1 is scored.
  • Retrievers and ASR included. The retrievers and live ASR are measured on the same NPU.
  • Full cost for PRISM. PRISM is charged the prefill and decode of every Verifier and Responder call, live ASR, and both retrievers. Each direct baseline is charged prefill plus a single output token per tick (e.g., a silent token).
methodOverallsubmissionon-device
Direct Qwen3.5-0.8B23.81.001.00
Direct Qwen3.5-2B27.71.581.64
Direct Qwen3.5-4B30.02.243.52
Direct Qwen3.5-9B34.63.365.00
Direct Qwen3-VL-8B36.33.165.25
Direct Qwen2.5-Omni-7B29.12.964.87
Decoder-gated 0.8B → 4B30.93.234.51
Decoder-gated 0.8B → 9B35.44.355.98
PRISM40.01.001.37

Table G1. Normalized energy on PhoStream (direct Qwen3.5-0.8B = 1.00), submission estimate vs. on-device measurement. Revised Table 1 reports the same values.

Table G1 recomputes Table 1 with the new on-device measurements.

  • The 3× claim holds and widens. On the device, larger models cost much more energy per call than the submission estimated, so the direct baselines rise sharply. PRISM runs the 0.8B Verifier on the retrieved candidates and calls the 4B Responder only 1.86 times per question, so even with its own decode, ASR, and retrievers now charged, its energy advantage over the best baseline, Qwen3-VL-8B, widens from 3.16× to 3.83×.
  • PRISM uses less energy and clearly outperforms the baselines. PRISM uses less energy than Qwen3.5-2B while scoring 12.3 points higher on PhoStream. It also scores 4.6 points higher than the decoder-gated two-stage baseline (0.8B → 9B) while using 4.36× less energy.

Accordingly, Appendix A.1 is rewritten around these measurements, and the new Fig. 7 plots the measured per-call energy of every model against input and output length. All other energy figures, Fig. 1(c), Fig. 5(a–b), and Fig. 8, are redrawn with the on-device measurements. The revisions in the manuscript are highlighted in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.