- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The core problem addressed in this paper is that resident AI assistants need to continuously monitor videos and conversation streams, but run large models at every step, and their battery capacity cannot withstand it. The author proposes the PRISM framework with the idea of doing cheap work all the time and doing expensive work as needed. Specifically, it is a three-stage pipeline: the first stage uses a lightweight encoder (22M parameter) to continuously search for "possibly relevant" candidates; The second level uses a validator of less than 1B, combined with context, to determine whether these candidates are "really worthy of action" and block false positives caused by "relevant but not yet timely"; It is only at the third level that the 4B model generates answers. In the experiment, PhoStream scored 40.0 overall, surpassing the best baseline. The most difficult Forward task increased from 7.7 to 22.7, premature response decreased from 81% to 9.4%, and energy consumption was only one-third of the best baseline.
Critical strengths
- The real pain point of edge deployment is that resident monitoring cannot run large models every step. The disassembly of retrieve verify respond separates expensive semantic judgments from continuous perception, with clear and reusable ideas, not just stack tricks.
- The Forward task increased from 7.7 to 22.7, premature response decreased from 81% to 9.4%, energy consumption decreased to one-third of the optimal baseline, and the main result, along with complete ablation and sensitivity analysis, provided a solid evidence chain.
Critical weaknesses
- The core selling point "Pareto improvement in performance energy consumption" is based on the assumption of fixed NPU power consumption and the calculation of model agents (using Qwen3 to impersonate Qwen3.5), and only calculates the prefill, which is purely a relative estimation rather than actual measurement. For a systematic paper that focuses on efficiency, this weakens the most weighty conclusion.
- The author admits that removing the verifier and allowing the 4B responder to make the final judgment directly increases the score by about 1.15 times. The high dependence of the benefits of the three-level cascade on conditions such as intervention sparsity is not universally valid, and the robustness of the method design is compromised.
Questions for authors
I have a few questions to confirm: Firstly, the energy consumption in Table 1 is based on the assumption of Qwen3 replacing Qwen3.5, fixed 1.75W power consumption, and only pre fill calculation. Have you ever verified it even on a small scale on real devices? If the agency relationship is not established, can the selling point of "3x power saving" still hold up? Secondly, you mentioned that after removing the verifier, the score actually increased by 1.15 times. So when is the three-level cascade really necessary? Have you ever considered having the 4B responder serve as both a gate control and a verifier? This directly affects my assessment of the robustness of the method. Thirdly, the visual vocabulary consists of COCO and Places365, with only about 600 semantic states. If the user's instructions exceed the scope of the vocabulary or if the language scene is changed, to what extent will the system degrade? Fourthly, SIR Bench only has 22 videos and 201 questions, and the dialogues and questions are still synthesized and filtered by Claude himself. With such a small scale, how strong conclusions do you think it can support?
6 comments
Rebuttal from Authors (2/3)
We give the details of each item below.
| Model | Prefill power | Decode power | Prefill energy | Energy per decoded token | Energy per call |
|---|---|---|---|---|---|
| Qwen3-0.6B | 1.00 | 0.66 | 0.92 | 1.39 | 1.27 |
| Qwen3.5-0.8B | 1.00 | 0.59 | 1.00 | 1.00 | 1.00 |
| Qwen3.5-2B | 1.23 | 0.56 | 1.78 | 1.78 | 1.78 |
| Qwen3-4B | 1.08 | 0.48 | 3.65 | 3.41 | 3.47 |
| Qwen3.5-4B | 1.14 | 0.50 | 3.84 | 3.38 | 3.50 |
| Qwen2.5-Omni-7B | 1.09 | 0.59 | 5.15 | 5.65 | 5.52 |
| Qwen3-8B | 1.07 | 0.56 | 5.73 | 6.09 | 5.99 |
| Qwen3.5-9B | 1.05 | 0.58 | 5.65 | 6.33 | 6.15 |
Table R3.3. Power and energy per model and per phase, normalized to Qwen3.5-0.8B (= 1.00), at 1,686 input tokens (the mean Verifier input) and 120 output tokens.
- Power is measured per model and per phase. As Table R3.3 shows, decode draws much less power than prefill for every model, only 0.44–0.66× of its prefill power.
- Decode draws less power, but its latency drives up the energy. Decoding takes much longer than prefill, so despite its lower power it accounts for 67% of a Verifier call's energy. It is now charged in full.
| Proxy pair | Output tokens | 1,024 in | 1,686 in | 2,048 in | 4,096 in |
|---|---|---|---|---|---|
| Qwen3-0.6B / Qwen3.5-0.8B | 1 | 0.95 | 0.92 | 0.97 | 1.15 |
| Qwen3-0.6B / Qwen3.5-0.8B | 120 | 1.00 | 1.27 | 1.07 | 1.38 |
| Qwen3-4B / Qwen3.5-4B | 1 | 0.99 | 0.95 | 1.02 | 1.09 |
| Qwen3-4B / Qwen3.5-4B | 120 | 0.94 | 0.99 | 0.99 | 1.08 |
Table R3.4. Per-call energy of each Qwen3 proxy relative to the corresponding Qwen3.5 model, measured on the same device (1.00 = identical).
- Qwen3 consumes nearly the same energy as Qwen3.5. As Table R3.4 shows, the ratios are close to 1.00, so the proxies in the submission did not deviate much from the actual deployment.
- For long inputs and outputs, Qwen3 consumes more energy. Owing to the linear attention of Gated DeltaNet, Qwen3.5-0.8B becomes cheaper than Qwen3-0.6B as sequences grow longer. The proxy therefore did not favor PRISM; if anything, it overestimated the Verifier's cost.
We have revised Appendix A.1 (rewritten, with the new Fig. 7), Table 1, Table 2, Sec. 4.2, the introduction's efficiency statement, and the energy figures (Fig. 1(c), Fig. 5(a–b), and Fig. 8) accordingly (in blue).
W2. The author admits that removing the verifier and allowing the 4B responder to make the final judgment directly increases the score by about 1.15 times. The high dependence of the benefits of the three-level cascade on conditions such as intervention sparsity is not universally valid, and the robustness of the method design is compromised.
Q2. Secondly, you mentioned that after removing the verifier, the score actually increased by 1.15 times. So when is the three-level cascade really necessary? Have you ever considered having the 4B responder serve as both a gate control and a verifier? This directly affects my assessment of the robustness of the method.
The three-stage cascade is necessary whenever the assistant monitors continuously, because it is what makes always-on monitoring affordable. Removing the Verifier, or letting the 4B Responder serve as both the gate control and the verifier, does raise the PhoStream score: the 4B Responder then makes the final decision on candidates the Verifier would have rejected, so fewer valid interventions are falsely rejected. This gain, however, requires running the 4B model on far more ticks: under the extended monitoring of Table 2, these configurations consume 2.0× and 5.5× the energy of PRISM.
| configuration | PhoStream score (× PRISM) | 4B Responder calls (% of ticks) | energy (× PRISM) |
|---|---|---|---|
| PRISM (Retriever → Verifier → Responder) | 1.00 | 10.2% | 1.0 |
| w/o Verifier (Retriever → Responder) | 1.15 | 36.9% | 2.0 |
| 4B Responder as both gate control and verifier | 1.26 | 100% | 5.5 |
Table R3.5. Score and cost of the three configurations. PhoStream score is the Overall score relative to PRISM, following the official protocol on the full benchmark; Responder call rates and energy are measured under the extended monitoring of Table 2 (25 videos, 2.6 hours of stream), with energy from the on-device measurements.
- The score gain comes from fewer false rejections. Without the Verifier, the 4B Responder also judges the candidates the Verifier would have rejected and recovers some valid interventions among them, which raises the score by about 1.15×.
- A 4B gate makes always-on monitoring prohibitively expensive. As Table R3.5 shows, the Verifier forwards only 10.2% of ticks to the 4B Responder, compared with 36.9% without it and 100% when the 4B Responder serves as its own gate. The energy grows accordingly, to 2.0× and 5.5× that of PRISM, far beyond the score gain of at most 1.26×, and the extra cost accumulates with every hour of monitoring, which an always-on assistant on a battery-powered device cannot sustain.
Rebuttal from Authors (3/3)
Q3. Thirdly, the visual vocabulary consists of COCO and Places365, with only about 600 semantic states. If the user's instructions exceed the scope of the vocabulary or if the language scene is changed, to what extent will the system degrade?
We thank the reviewer for this interesting question. PRISM does not require the requested concept to be in the vocabulary. The Retriever wakes the Verifier by embedding similarity, so an instruction about a missing concept enrolls its nearest concepts instead, and frames showing the requested concept still match them.
To test this, we removed the requested concept from the vocabulary and checked whether its images still wake the Verifier. Table R3.6 shows examples, and Table R3.7 repeats this for every COCO and Places365 concept.
| removed concept | concepts enrolled instead | wake recall: in vocabulary → removed |
|---|---|---|
| elephant | horse, cow, bear, giraffe, sheep | 98% → 90% |
| pizza | pizzeria, sandwich, fast food restaurant, restaurant, delicatessen | 95% → 93% |
| bookstore | indoor library, gift shop, outdoor library, drugstore, book | 97% → 93% |
Table R3.6. Examples with the requested concept removed from the vocabulary (e.g., "Tell me when you see an elephant"). Wake recall is the share of images containing the concept that wake the Verifier.
| requested concept | COCO wake recall | false-wake rate | Places365 wake recall | false-wake rate |
|---|---|---|---|---|
| in vocabulary | 68.0% | 6.8% | 84.2% | 4.8% |
| removed | 56.8% (0.84×) | 7.7% | 70.6% (0.84×) | 5.3% |
| removed with its supercategory | 51.1% (0.75×) | 7.0% | – | – |
Table R3.7. Wake recall on images containing the requested concept and false-wake rate on images without it, averaged over all 80 COCO and 365 Places365 concepts and two instruction templates.
- The system degrades gracefully. As Tables R3.6 and R3.7 show, without the requested concept PRISM keeps 0.84× of its wake recall on both datasets (0.75× even with all related concepts removed), while false wakes change by less than one percentage point.
A missing concept can also be added by precomputing a few text embeddings, with no retraining. On the language side, the State Retriever is never trained on PhoStream, so all PhoStream results already come from conversations outside its training distribution.
We have added this analysis to Sec. 4.2 (new Table 3, in blue).
Q4. Fourthly, SIR Bench only has 22 videos and 201 questions, and the dialogues and questions are still synthesized and filtered by Claude himself. With such a small scale, how strong conclusions do you think it can support?
We thank the reviewer for this question. SIR-Bench supports a focused conclusion: PRISM's Retriever and Verifier make specific, relevant decisions about an implicit need for help, which raise accuracy from 60.7% to 65.2% with the same Responder (Table 5). No intervention-timing or energy claim in the paper relies on it.
- Real requests may reflect implicit needs. PhoStream specifies the target through explicit cues such as "Wait until …," which users rarely say in daily life. A user is more likely to mention feeling tired and unwell, and a helpful assistant should then point to a nearby place to sit with water (Fig. 6). SIR-Bench is built so that a model cannot answer without inferring this unstated need.
- It tests whether the Retriever and Verifier make specific decisions. A useful decision is not a generic "the user needs help" but a concrete state, such as physical discomfort or the user's current surroundings, that points to the right intervention. With the same 4B Responder, adding the retrieved user states and the Verifier summary raises accuracy by 4.5 points.
To verify this filtering beyond Claude, we gave model families that played no role in construction only the question and the four options (Table R3.8).
| model | accuracy without input |
|---|---|
| GPT-5.5 | 30.8% |
| GPT-5.4-mini | 21.4% |
| Gemini-3.5-Flash | 25.4% |
| Gemini-2.5-Flash | 30.3% |
| Qwen3.5-4B (submission) | 25.4% |
Table R3.8. Accuracy when only the question and the four options are given (chance = 25%). Same prompt and option order as the submission.
- SIR-Bench cannot be solved without the input. As Table R3.8 shows, every model stays close to chance (21–31%) without the input, whereas Qwen3.5-4B reaches 60.7% with the video and conversation.
- The filtering holds beyond Claude. GPT, Gemini, and Qwen models all stay near chance, so the filtering is not specific to the generator.
We have clarified the intent of SIR-Bench in Sec. 4.3, and its construction together with the no-input check across model families in the Appendix (in blue). Inferring users' implicit needs from everyday context is an important yet underexplored challenge; we see SIR-Bench as a starting point for this direction and plan to expand it, guided by community feedback.
Rebuttal from Authors (1/3)
We thank Reviewer n2M4 for the careful review and the concrete questions. We deployed the actual Qwen3.5 models and the baselines on a Samsung Galaxy S26 Ultra and measured their energy on the device, and we evaluated the 4B Responder as its own gate and verifier on a continuous timeline.
We have now measured the Table 1 models on a real device, not only on a small scale, and the 3× claim holds and widens: PRISM uses 0.26× the energy of the best baseline, Qwen3-VL-8B (submission estimate: 0.32×). Qwen3.5 could not yet be deployed on the device at submission time because of its new architecture; it now can. We measured the energy of the models on a Galaxy S26 Ultra and recomputed Table 1 (Table R3.1).
Table R3.1. Normalized energy on PhoStream (direct Qwen3.5-0.8B = 1.00), submission estimate vs. on-device measurement. The revised Table 1 reports the same values.
Table R3.2 summarizes how each assumption of the submission is replaced.
Table R3.2. Each assumption of the submission's Appendix A.1 and its on-device replacement.