acceptodds
Under review as a conference paper at ICLR 2027

Exact Where It Matters: Downstream-Guided Attention Refinement for Long-Video Understanding

Abstract

Approximating attention accelerates long-video understanding, but local error alone does not determine whether a correction improves the answer. We introduce EWiM (Exact Where it Matters), a training-free method for downstream-guided attention refinement during offline video–question prefilling. At selected layers, EWiM probes a bounded candidate pool through isolated corrections on sampled query rows and measures their effects on a question-conditioned next-layer readout at the terminal prompt position. Incremental updates reuse the common coarse state and refresh only the changed contributions to the next-layer readout. An execution-aware allocator uses these one-step sensitivity scores and hardware-profiled marginal costs to select full-block refinements, updating costs to account for shared row computation within a budget that includes probing overhead. Input tokens, model parameters, and the decoding procedure remain unchanged. Across eight main model–benchmark pairs, EWiM retains 99.2% of dense accuracy on average at a 1.46× geometric-mean speedup in time to first token, including visual encoding and all acceleration overhead. Controlled interventions show that the selected block sets yield higher signed answer-margin utility than those chosen by local-error controls. Further experiments demonstrate transfer to additional backbones and composition with token compression.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-video understanding, Video large language models, Attention refinement, Adaptive computation, Efficient inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zjhQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes EWiM, a training-free method for accelerating the prefill stage of Video LLMs on long videos. EWiM first approximates most visual attention blocks cheaply by pooling, and then selectively recomputes exact attention for a small set of blocks. Blocks are chosen by how much correcting them changes the representation of the question’s final token, and are selected under a latency budget using a hardware-aware cost model. Across four video benchmarks and multiple Video LLMs, EWiM retains nearly all of dense accuracy at a moderate time-to-first-token speedup, and it can be combined with token compression.

Critical strengths
  1. A well-posed question within approximate attention. The paper argues that attention interactions should be prioritized by their effect on the answer rather than by local approximation error, and controlled comparisons against local-error and text-aware signals support this choice. This offers a useful principle for deciding where exact computation matters in sparse attention.

  2. Realistic and broad evaluation. All overheads (probing, selection, and refinement) are included in the reported latency, and the cost model accounts for computation shared among blocks. The evaluation covers four benchmarks on full splits, a wide range of token-compression and sparse-attention baselines, and multiple backbones. Together, these make the reported accuracy–latency trade-off credible.

Critical weaknesses
  1. The writing makes the method very hard to understand. Section 3 relies heavily on undefined, implementation-level terminology and compressed notation. The underlying idea is simple: approximate most attention blocks by pooling, then recompute exactly the few blocks that most affect the question’s final token. Yet it could be conveyed far more clearly with a standard attention formulation, a notation table, and a step-by-step description. This is consequential to my assessment because the presentation makes it difficult to verify the method and to judge its contribution.

  2. The practical motivation relative to existing efficiency strategies is unclear. FlashAttention already removes the memory bottleneck of exact attention, and token compression reduces attention, feed-forward, and KV-cache costs together while remaining compatible with such kernels. EWiM, by contrast, reduces only attention computation, increases peak memory, and leaves the KV cache unchanged, while yielding a speedup comparable to token compression. It also requires many components and hyperparameters, and much of its latency budget is spent repairing a lossy coarse approximation. This is the main reason for my rating: the paper does not show when its added complexity provides benefits that simpler existing strategies cannot.

Questions for authors
  1. In which settings does EWiM offer benefits that token compression combined with efficient exact attention (e.g., FlashAttention) cannot? Can you provide a latency breakdown (vision encoding, attention, feed-forward) across context lengths?
6 comments
AuthorsOpenReview ↗

Reply 1: Measured trade-offs against token compression and runtime across context lengths

EWiM preserves the sequence supplied to it and approximates selected visual-attention interactions. It allocates additional computation to corrections with stronger sampled effects on the current question’s next-layer readout. Its target is attention computation during prefilling, rather than reducing token count or KV storage.

Comparison with optimized baselines. Dense and FlashVID use the same optimized exact-attention backend for retained-token computation. EWiM uses kernels suited to mixed proxy attention, sampled probing, and block refinement. The revised implementation table identifies the kernels, compilation settings, and fallback behavior. Any auxiliary computation required by a method remains included in its measured cost.

Accuracy uses the full benchmark split. Mean TTFT uses the same fixed 200-request timing panel across methods within each model–benchmark pair, with common repeat and aggregation procedures. Development videos are source-disjoint from evaluation videos.

On Qwen2.5-VL/Video-MME, the highest-accuracy observed configuration of each baseline within ±2% of EWiM’s mean TTFT is:

MethodAccuracy (%)Mean TTFT (s)
EWiM61.02.16
VecAttention60.02.17
FlashVID + optimized exact attention59.62.15

The observed differences are +1.0 and +1.4 percentage points (pp) at comparable TTFT. These are evaluation-selected descriptive comparisons. Conditional on the displayed configurations, source-video-clustered paired 95% intervals are [0.5, 1.5] and [0.8, 2.0] pp; these pointwise intervals are not adjusted for operating-point selection.

The development-frozen primary comparisons and actual latencies are reported separately for all eight main model–benchmark pairs. Searches include 15 EWiM configurations, 12 VecAttention configurations, and FlashVID visual-token retention of 40–100%, with native parameter values and fixed settings disclosed. This is a specified search range, not FlashVID’s complete native range. The curves retain points outside the matching window, including VecAttention’s faster 58.6% / 1.65 s configuration. The evidence supports the reported high-fidelity trade-offs, not uniform dominance over compression or sparse attention.

Runtime across context lengths. Revised Figures 6–7 provide numerical profiles over 8K, 16K, 32K, and 64K alongside independently measured deployment-path TTFT. The cross-method functional view compares Dense, FlashVID, and EWiM through vision encoding, decoder attention, FFN, projection/normalization, and other work. The EWiM stage view separately reports Coarse, Probe, Select, Exact refinement, and Refresh.

These are alternative classifications, not additive categories. For example, probing’s FFN work appears under FFN in the functional view and Probe in the stage view. Row updates are counted once, reused projections are not charged twice, and kernels that cannot be reliably decomposed are identified separately.

Length groups are defined by the common pre-compression host-formatted request length, NorigN_{\mathrm{orig}}. Methods use the same requests within each timing group; FlashVID’s retained length is reported separately. Total tokens, visual tokens, retained visual tokens, and video duration are distinguished.

Component profiles explain computational work; independent TTFT measures request latency. Asynchronous execution, overlap, fusion, and profiling boundaries can prevent component times from summing to TTFT. Residual or unattributed wall time is disclosed where applicable rather than assigned arbitrarily to attention. Real requests support task-quality conclusions; fixed-shape replay isolates execution effects.

The decomposition explains the execution conditions behind the measurements. A large Dense attention fraction alone does not establish superiority over compression, and the aggregate high-fidelity comparison is not presented as evidence of dominance at every context length.

AuthorsOpenReview ↗

Reply 2: Refinement overhead, calibration, composition, and deployment limits

Net benefit of refinement. Four measured Qwen2.5-VL/Video-MME operating points quantify the cost of refinement:

ConfigurationAccuracy (%)Mean TTFT (s)
Dense61.33.20
Coarse54.31.88
Denser Single Pass60.12.17
EWiM61.02.16

Relative to Coarse, EWiM gains 6.7 percentage points (pp) with a 0.28 s increase in TTFT. Relative to Dense, it saves 1.04 s with a 0.3-pp accuracy reduction. Compared with Denser Single Pass, its observed accuracy is 0.9 pp higher at comparable measured TTFT.

The 0.28 s is a net difference between independent complete runs; component costs are measured separately. Denser Single Pass enlarges the base exact set, retains other pooled contributions, and performs no DGP. It tests spending additional computation directly on base exact attention rather than feedback-guided refinement. Its 132 additional full blocks, versus EWiM’s 84, are block counts. EWiM also incurs sampled probing, readout, and refresh work, so this ratio is not a total-compute or FLOP ratio.

Composition with compression. With the same fixed 50% FlashVID-compressed input on this model–benchmark pair:

Attention policyAccuracy (%)Mean TTFT (s)
Dense58.81.95
EWiM60.02.15

EWiM gains 1.2 pp at an additional 0.20 s. This fixed-input operating point improves accuracy at higher latency; it does not demonstrate acceleration over Dense on those tokens. Nor does it establish recovery of information removed by merging or greater similarity to uncompressed Dense outputs.

The separate joint-selection results report retention, accuracy, actual TTFT, and the configuration-selection protocol for each policy. Combination speedups are measured directly. In this setting, EWiM preserves the compressed sequence it receives, rather than the original pre-compression token sequence.

Configuration and calibration. Training-free does not imply calibration-free. The revised configuration table separates fixed design settings, development-selected operating points, host adapters, and hardware profiles. The reference uses t=64t=64, r=4r=4, Cℓ=256C_\ell=256, a 50:50 ranked/coverage mixture, and probing microbatch size 32. The selected reference operating point uses k0=16k_0=16 and λB=0.25\lambda_B=0.25; both are included in the disclosed development search.

The layer budget is

Bℓ=λBt^dense,ℓ,B_\ell=\lambda_B\widehat t_{\mathrm{dense},\ell},

where the profile is Dense attention time for that layer and input shape, not full-model TTFT. Host-specific layer indices and execution settings are listed in the configuration table.

Calibration uses 220 replay configurations per host–backend–GPU combination without answer labels. The primary H800 calibration takes 41 minutes, including preparation, compilation, warm-up, and repeated measurements. This is an offline cost for the identified combination, separate from per-request TTFT. On A100, coefficients are recalibrated and budgets recomputed; accuracy and TTFT are re-evaluated because accepted sets can change.

Memory and deployment limits. In the reported H800 setting at 64K, allocated GPU memory is 41.6 GiB for EWiM and 38.2 GiB for Dense. The 3.4-GiB difference is a total allocated-memory difference, not an isolated temporary-workspace measurement.

EWiM retains ordinary token-indexed KV storage and the ordinary FFN work associated with retained tokens. Refinement changes intermediate states and potentially KV contents, and adds row-update work. KV capacity, contents, and cross-question reusability are distinct properties.

For three and eight independent questions about the same video, EWiM’s cumulative TTFT exceeds Dense’s, while VecAttention remains faster. This measured limitation rules out extending the single-request speedup to those cached workloads.

The evaluated settings are batch-one prefilling on H800 and the recalibrated A100 configuration. Primary single-request timing uses no cross-request visual-feature or KV reuse. TTFT runs from decoded frames in host memory to first-token logits and includes visual encoding and all online method overhead. Larger batches, concurrent serving, multi-GPU execution, and decoding acceleration remain outside the evaluated scope.

AuthorsOpenReview ↗

Reply 3: A clearer formulation and execution sequence for Section 3

Section 3 now follows the attention computation and execution order: a provisional mixed-attention pass, isolated sampled probes, cost-aware selection, and joint full-block refinement. A compact notation table defines the symbols before use. The revised algorithm gives this sequence explicitly; detailed state lifecycles and kernels remain in the appendix.

1. Shared normalization. For one layer and query head, let aij=qi⊤kj/da_{ij}=q_i^\top k_j/\sqrt d. A KV tile JJ contributes

ziJ=∑j∈Jieaij,niJ=∑j∈Jieaijvj,z_{iJ}=\sum_{j\in J_i}e^{a_{ij}},\qquad n_{iJ}=\sum_{j\in J_i}e^{a_{ij}}v_j,

where JiJ_i contains keys visible to query ii under the host mask. Tildes denote the corresponding pooled proxy contributions. With exact tiles EiE_i and proxy tiles PiP_i,

oi=∑J∈EiniJ+∑J∈Pin~iJ∑J∈EiziJ+∑J∈Piz~iJ.o_i=\frac{\sum_{J\in E_i}n_{iJ}+\sum_{J\in P_i}\widetilde n_{iJ}}{\sum_{J\in E_i}z_{iJ}+\sum_{J\in P_i}\widetilde z_{iJ}}.

Refinement replaces a block’s numerator and denominator contributions under this shared normalization. Selected contributions are computed exactly, but remaining proxies still affect the denominator; the resulting row is not a fully Dense output.

A candidate block comprises a query head, a query tile, and a KV tile. Approximation is restricted to entirely visual blocks whose entries are all visible under the host mask. Visible text-related and causal-boundary interactions remain exact; masked entries contribute zero. Pooling retains its multiplicity correction and the native positional and query-to-KV-head conventions.

2. Incremental downstream probing. Each candidate trial starts from the same provisional state and changes only sampled visual rows. The terminal nonvisual prompt position is unchanged, so its next-layer query is fixed. EWiM refreshes the changed rows’ native next-layer KV projections, updates the terminal attention row from shared baseline statistics, and evaluates that row’s output projection and feed-forward operations. It does not rerun the remaining decoder for each candidate.

The terminal query is shared; the changed keys and values are not frozen. DGP uses the normalized response magnitude defined in Eq. (5), without answer labels. Its scoring definition is unchanged by the presentation rewrite.

3. Allocation by conditional execution cost. Let AA be the accepted set and ΔU=U(A∪{b})−U(A)\Delta U=U(A\cup\{b\})-U(A) the newly affected query-row work, deduplicated across heads. The fitted marginal cost is

c^ℓ(b∣A)=αℓ+βℓΔU+γℓ1[A=∅].\widehat c_\ell(b\mid A)=\alpha_\ell+\beta_\ell\Delta U+\gamma_\ell\mathbf 1[A=\varnothing].

A block reusing already affected rows avoids additional row-update work; new rows and startup incur the corresponding costs. ECA selects the feasible candidate with the largest sensitivity-to-marginal-cost ratio, keeps sensitivity scores fixed, and updates costs after each acceptance. It stops when no remaining candidate fits the profiled budget, which includes probing and allocation overhead. This models execution sharing, not additive answer utility.

4. Joint full-block refinement. Probe states are discarded rather than accumulated. Once selection ends, every accepted block is recomputed over its full query tile, including sampled rows. Numerator and denominator corrections are accumulated within each head. EWiM then assembles head outputs, performs one shared native update per affected row, refreshes dependent projections, and commits the state before ordinary next-layer execution. Shared row updates therefore do not mix the softmax statistics of different heads.

5. Offline routing and reuse. Attention contributions obey the host mask, while routing uses the observed full video–question prompt. Question-dependent selection can change visual-prefix states, so mask compliance does not imply prefix invariance or unrestricted cross-question reuse.

The overview distinguishes unchanged model weights, decoding procedure, and supplied token sequence from the attention computation and intermediate states that refinement changes. The formulation, algorithm, and execution description now use the same notation and state sequence.

Reviewer_zUE7OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EWiM, a training-free attention refinement method for offline video-question prefilling. Starting from a mixture of exact and pooled visual attention, it probes candidate blocks through sampled-row exact corrections. Candidates are ranked by their effect on the terminal prompt state after the next decoder layer. An execution-aware allocator combines these sensitivity scores with profiled marginal costs, accounting for row-update reuse between selected blocks. Accepted blocks are recomputed in full and committed jointly. Across two primary backbones and four benchmarks, the method reports 99.2% average accuracy retention relative to dense attention and a 1.46x geometric-mean TTFT speedup. Additional experiments study signed answer-margin utility, transfer to two more backbones, token-compression composition, and repeated-question workloads.

Critical strengths
  • The paper goes beyond local attention approximation error by evaluating whether selected refinements matter downstream. The controlled signed-utility audit reports higher mean answer-margin utility for DGP than Local Error or Text-Aware Local Error, with a shuffled-score control removing the benefit. This is useful evidence for the proposed ranking signal, while the paper appropriately distinguishes response magnitude from correctness.
  • The execution design and evaluation are reasonably comprehensive. TTFT includes visual encoding and refinement overhead, the allocator explicitly models shared row work, and transfer retains the scoring/candidate settings across additional hosts. The ablations also compare against a denser single pass at similar latency. Together these results support a practical accuracy-latency benefit rather than only a theoretical reduction in attention work.
Critical weaknesses
  • The main accuracy-latency frontier is less fully characterized than the presentation suggests. Figure 4 shows selected operating points and explicitly omits per-method budget sweeps, although Appendix B.3 describes development selection at three targets. Some mechanism gains are modest, such as 61.0 versus 60.6 for dynamic versus static cost allocation and net gains of 34 versus 31 repaired questions over Text-Aware in the 500-question LVB audit. The appendix describes bootstrap and routing-seed analysis, but does not report paired uncertainty for these central accuracy differences. Full measured curves and intervals are important for determining how robust the incremental advantage is.
  • The practical scope is narrower than a general long-video acceleration claim. Profiling and main runtime evaluation use one H800 at batch size one; the method retains ordinary token-indexed KV storage and adds workspace (41.6 versus 38.2 GiB at 64K). Question-dependent refinement also limits prefix reuse, and Appendix C.7 shows faster cumulative growth than VecAttention under repeated questions. These limitations are acknowledged, but they make the missing breakdown of cost-model prediction errors and transfer across execution conditions consequential for deployment.
Questions for authors
  1. Why should a large downstream response identify a beneficial correction, rather than merely an influential one? The score is an unsigned next-layer change, whereas the desired outcome is improved final-answer quality. Table 2 usefully shows positive average signed utility, but some selected sets still have negative utility, and the complete-run LVB audit reports more regressions for EWiM than for the local controls.

  2. Does selecting several individually influential blocks preserve their joint usefulness? Candidate scores are measured as isolated sampled interventions from a common coarse state and then kept fixed, while accepted blocks interact through shared normalization and subsequent nonlinear computation. The signed-set audit already measures joint utility and should not be replaced by a sum of isolated effects.

4 comments
AuthorsOpenReview ↗

Title: Reply 1: Measured accuracy-latency curves and paired uncertainty

We thank the reviewer for highlighting the need to characterize the operating trade-off and uncertainty in the incremental gains.

Measured curves and selection protocol. Revised Figure 4 and Table R1 report measured configurations for all eight main model–benchmark pairs. Accuracy uses the full benchmark split; mean TTFT uses the same fixed 200-request panel across methods within each pair, with identical timing and aggregation procedures. TTFT includes visual encoding and all method-specific overhead. Development videos are source-disjoint from evaluation videos.

The searches cover all 15 combinations of k0∈{8,16,32}k_0\in\{8,16,32\} and λB∈{0,0.1,0.25,0.5,1}\lambda_B\in\{0,0.1,0.25,0.5,1\} for EWiM, 12 VecAttention configurations, and FlashVID retention ratios of 40–100%. Table R1 specifies the native parameter values and fixed settings. All valid measured points are retained; OOM and execution failures are reported separately.

Primary configurations are development-selected and frozen before evaluation. Table R1 gives both accuracies, both actual mean TTFTs, signed differences, and paired intervals for all eight VecAttention comparisons. Seven accuracy point estimates favor EWiM; the remaining absolute difference is at most 0.2 percentage points (pp). Seven positive estimates are not interpreted as seven statistically significant gains.

As a descriptive example on Qwen2.5-VL/Video-MME, the highest-accuracy observed configuration of each baseline within ±2% of EWiM’s mean TTFT is:

MethodAccuracy (%)Mean TTFT (s)
EWiM61.02.16
VecAttention60.02.17
FlashVID + optimized exact attention59.62.15

These points are evaluation-selected, not development-frozen. The full curves retain configurations outside this window, including VecAttention’s faster 58.6% / 1.65 s point. We characterize the measured high-fidelity trade-off without claiming uniform frontier dominance.

ECA versus Static Cost. Independent end-to-end runs share inputs, frozen non-allocation settings, scoring and candidate-construction rules, routing seeds, and the budget protocol. Different refinements can produce different subsequent states and scores.

ECA achieves 61.0% at 2.16 s, versus Static Cost’s 60.6% at 2.15 s. The paired difference is +0.4 pp, with a source-video-clustered 95% interval of [0.1, 0.7] pp, using consistent routing-seed aggregation. This supports a modest gain for this frozen configuration. Actual additional-cost and budget-usage measurements accompany the comparison; similar complete-system TTFTs do not establish identical refinement work.

Complete-run LVB audit. Changes relative to the common Coarse baseline on 500 questions are:

SignalRepairsRegressionsNet
Local Error35629
Text-Aware Local Error38731
DGP / EWiM441034

For the execution represented by these counts, DGP produces six more repairs and three more regressions than Text-Aware, yielding three additional correct answers (+0.6 pp). The net-repair difference and final accuracy difference are the same outcome, not independent evidence.

Table R2 identifies the run/seed composition of each estimate and reports the paired task-accuracy point estimate and 95% interval. Direct discordant-pair counts are tied to their corresponding binary prediction records; multi-seed estimates are distinguished from the integer decomposition above.

The bootstrap resamples source videos as paired clusters, retaining all their questions and recomputing the question-weighted accuracy difference. Routing seeds are aggregated within questions using the same rule as the accompanying point estimate, rather than counted as additional independent questions. The +0.6 pp arithmetic from the displayed counts is not substituted for a differently aggregated estimate.

Accuracy uncertainty, routing-seed variability, and timing repeatability are reported separately. A statistically unresolved difference is not evidence of equivalence, and pointwise intervals do not provide a simultaneous confidence guarantee for the entire frontier.

AuthorsOpenReview ↗

Reply 2: Downstream sensitivity and joint refinement utility

As defined in Eq. (5), DGP measures an unsigned next-layer response to an exact/proxy intervention. It ranks downstream sensitivity, not signed answer improvement. A larger response identifies greater sensitivity of the question-conditioned readout to that approximation choice; its magnitude does not determine whether the final answer improves. The value of prioritizing these corrections is evaluated empirically.

Inference-time DGP scoring and selection do not access correct-answer labels. Labels are used for development-set operating-point selection and offline utility evaluation; hardware calibration is label-free.

Controlled joint-set utility. For question qq at layer ℓ\ell, we measure

Jqℓ(A)=mq(A;π0)−mq(∅;π0),\mathcal J_{q\ell}(A)=m_q(A;\pi^0)-m_q(\varnothing;\pi^0),

where AA is the jointly applied refinement set and π0\pi^0 fixes subsequent discrete routing. Correct and competing answer strings are fixed before intervention. All dependent hidden states and caches are recomputed after the full-block corrections are applied jointly. The measurement includes shared normalization and subsequent nonlinear propagation, conditional on these routes.

Utility is in nats per answer token. Positive and negative sets satisfy J>10−3\mathcal J>10^{-3} and J<−10−3\mathcal J<-10^{-3}; the remainder are neutral.

SignalMean J\mathcal JPositive (%)Neutral (%)Negative (%)
Local Error+0.3258.228.313.5
Text-Aware+0.4564.524.311.2
DGP+0.6878.115.56.4
Shuffled DGP−0.0520.354.325.4

Table R3 reports the DGP–Text-Aware and DGP–Local Error paired mean-utility contrasts and their 95% interval endpoints, computed from the underlying unrounded records. Comparisons share questions, answer strings, intervention states, and subsequent routes. Layers and routing seeds use the same aggregation weights as the reported means; source videos are resampled as paired clusters. The intervals concern average signed answer-margin utility, not task accuracy or a per-block probability of benefit.

For shuffled DGP, incoming states, candidates, sampled rows, the ECA executor, cost definitions, and resource constraints are fixed; only score–candidate correspondence changes. Accepted blocks, affected rows, reuse, and execution time are allocator outputs, whose measured values are reported alongside utility. The question-transfer control transfers quota-matched block selections and recomputes target-question states and caches.

Isolated scores and joint outcomes. Candidate scores are computed from a common provisional state and remain fixed during allocation. ECA updates execution costs as row reuse changes; it does not assume

J(A)=∑b∈AJ({b}).\mathcal J(A)=\sum_{b\in A}\mathcal J(\{b\}).

The joint audit measures the selected set directly, including its interactions, rather than summing isolated effects. It establishes the observed set-level effect under fixed routes. Complete runs separately evaluate the deployed policy, including subsequent adaptive-routing feedback.

The separate LVB audit reports 44 repairs and 10 regressions for DGP, versus 38 repairs and 7 regressions for Text-Aware. The 6.4% negative-set rate in the controlled audit and the complete-run regressions have different units and outcomes. Thus, fewer negative-utility sets in the controlled audit do not imply fewer answer regressions in deployment.

These results support the measured average joint value of one-shot selection while retaining its negative outcomes. They do not establish global optimality or quantify the gap to sequential re-scoring. Such re-scoring would remain an unsigned-score heuristic and would require accounting for its additional probing and state-update costs.

Sampling approximation. Table 7 reports sampled/full-row Kendall τb=0.71\tau_b=0.71–0.740.74 for DGP, with an accompanying r=1,2,4,8r=1,2,4,8 analysis. Each unshuffled signal is compared with its own full-row counterpart. This assesses fidelity to that signal’s full-row ranking, not correlation with signed answer utility. Sampling fidelity, controlled joint utility, and complete-run accuracy address distinct parts of the method’s evidence.

AuthorsOpenReview ↗

Reply 3: Cost-model validation and deployment scope

We added independent cost-prediction validation and evaluated a second GPU after recalibration. These measurements extend the execution evidence while preserving the distinction between single-request acceleration and repeated-query reuse.

Prediction and hardware transfer. Calibration and validation trajectories are disjoint. For the two primary hosts under the reported H800 deployment profiles, total refinement-time median/P90 absolute percentage errors (APE) are 4.2%/9.1%; on shapes excluded from fitting, they are 7.1%/13.8%. For measured marginal costs ≥5 ms, median/P90 APE are 8.4%/17.6%. Below 5 ms, median/P90 absolute errors are 1.4/4.2 ms. Small-cost cases remain included in the budget analysis.

MeasurementH800Recalibrated A100
Total refinement-time median APE4.2%7.4%
Probed layer-execution means exceeding budget1.3%3.8%
Arithmetic-mean accuracy retention, eight main pairs99.2%99.1%
Geometric-mean TTFT speedup1.46×1.42×

Budget excess is assessed using the repeated-measurement mean of each final ECA layer execution that invokes probing, including probes accepting no blocks. Layers bypassed before probing are excluded and counted separately. These are layer-level statistics, not request-level deadline-miss rates.

On A100, cost coefficients are recalibrated and budgets recomputed while scoring, candidate construction, k0k_0, λB\lambda_B, and probing layers remain fixed. Both accuracy and TTFT are re-evaluated because changed costs can change accepted sets. Each GPU’s speedup uses its own Dense reference. The budget-excess rates and speedups are separate measurements, without an inferred causal relationship.

Calibration uses 220 replay configurations per host–backend–GPU combination, without answer labels. The primary H800 calibration takes 41 minutes, including preparation, compilation, warm-up, and repeated measurements. The calibration table identifies this combination and reports the others separately. Cost and budget tables specify probing, selection, exact refinement, and refresh accounting.

Memory and retained computation. At 64K, allocated GPU memory is 41.6 GiB for EWiM and 38.2 GiB for Dense. These are total allocated-memory measurements; the 3.4-GiB difference is not presented as an isolated temporary-workspace measurement.

EWiM retains the ordinary token-indexed KV footprint. Refinement can nevertheless change KV contents and cross-question reusability. Retained tokens also retain their ordinary FFN work, while refinement adds row-update computation. The method does not reduce KV storage.

Repeated questions. These workloads contain independent questions about the same video, not accumulated multi-turn conversation. The revised Figure 8 results table reports method accuracy and cumulative TTFT on the same videos and questions within each workload. Cache construction, reads, transfers, and refresh are timed; retained-cache capacity and allocated memory are reported separately.

At three and eight questions per video, EWiM’s cumulative TTFT exceeds Dense’s, while VecAttention remains faster. Cache-consistency checks do not establish a cross-method efficiency advantage, and no unmeasured query-count crossover is inferred.

Our validated execution settings are batch-one prefilling on H800 and the recalibrated A100 configuration. Primary single-request measurements use no cross-request visual-feature or KV reuse. TTFT starts with decoded frames in host memory and ends at first-token logits, including visual encoding and all acceleration overhead. T(M)/MT(M)/M denotes amortized first-token latency.

The single-request acceleration does not extend to the tested three- and eight-question cached workloads. Larger batches, concurrent execution, multi-GPU serving, throughput gains, and decoding acceleration remain outside the evaluated scope.

Thanks for the feedback! My concerns have been addressed. I lean to accept.

Reviewer_GhyUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EWiM is a training-free framework for allocating exact attention computation in offline long-video QA. Downstream-Guided Probing evaluates sampled-row corrections through a question-conditioned next-layer readout, and Execution-Aware Compute Allocation selects block refinements using marginal costs that account for shared row computation. Across two main 7B models and four benchmarks, the paper reports 99.2% relative accuracy retention at a 1.46x geometric-mean TTFT speedup.

Critical strengths
  1. DGP addresses a meaningful distinction between local approximation error and downstream relevance. Applying candidate corrections and observing their effects on a question-conditioned representation provides a concrete sensitivity signal. The incremental readout avoids repeatedly executing the entire decoder.

  2. ECA connects allocation to execution by accounting for reusable output projection, feed-forward computation, and projection refresh when blocks share query rows.

  3. The presentation distinguishes response magnitude from answer improvement. Signed answer-margin interventions, shuffled scores, and question-transfer experiments provide informative controls. The primary TTFT measurements include visual encoding and method overhead, and repeated-question limitations are acknowledged.

Critical weaknesses
  1. The accuracy-latency advantage over strong baselines remains unclear. Table 1 reports 99.2% relative accuracy at 1.46x speedup for EWiM, versus 95.3% at 1.89x for VecAttention. These different operating points do not establish the better trade-off. Appendix B.3 describes searches under common latency targets, but Figure 4 omits per-method budget sweeps. Matched-latency or matched-accuracy comparisons are needed to establish the relative trade-off across operating points.

  2. ECA's cost predictions lack direct validation. The paper does not quantify prediction errors against measured execution times, calibration data and time, or recalibration requirements across models and hardware. Table 9's 0.4-percentage-point gain over Static Cost at nearly identical latency supports the selection policy, but does not establish prediction reliability. The exact probing-layer indices and mapping from the searched budget parameter to per-layer budgets also remain insufficiently specified.

  3. The practical benefit is sensitive to query reuse. In Figure 8, EWiM's cumulative first-token latency exceeds Dense for three and eight questions per video when compatible prefix computation is reusable, while VecAttention remains faster. This acknowledged limitation should more clearly delimit the intended deployment scenarios and the quality-latency trade-off for repeated queries.

Questions for authors
  1. Can the authors show measured accuracy-latency curves (using TTFT as latency) from the existing budget searches, particularly for VecAttention, and specify each method's search range? How does EWiM compare at matched TTFT or matched accuracy?

  2. What are ECA's execution-time and marginal-cost prediction errors, calibration requirements, and exact probing-layer and per-layer budget settings?

3 comments
AuthorsOpenReview ↗

Reply 1: Measured accuracy-latency trade-offs

We thank the reviewer for noting that the selected points in the original Figure 4 did not establish the relative accuracy–TTFT trade-off. The expanded figure and search tables now report measured configurations for all eight main model–benchmark pairs.

Protocol and search coverage. Accuracy uses the full benchmark split; mean TTFT uses the same fixed 200-request panel for every method within each pair, with identical repeat and aggregation procedures. Development videos are source-disjoint from evaluation videos. TTFT includes visual encoding and all method-specific overhead. Hardware, precision, and timing procedures are matched. Dense and FlashVID use the same verified optimized exact-attention backend, identified by kernel and version in the revised configuration table.

EWiM evaluates all 15 combinations of k0∈{8,16,32}k_0\in\{8,16,32\} and λB∈{0,0.1,0.25,0.5,1}\lambda_B\in\{0,0.1,0.25,0.5,1\}. VecAttention evaluates 12 native-control configurations spanning aggressive sparsity to high-fidelity operation; FlashVID covers 40–100% retention. The search tables enumerate the parameter values, fixed settings, and operating points. All valid measured configurations are shown; OOM and execution failures are reported separately and excluded from complete-workload comparisons.

Development-frozen comparisons. Primary configurations are selected on development data and frozen before evaluation. Across the eight VecAttention comparisons, seven accuracy point estimates favor EWiM; the remaining absolute difference is at most 0.2 percentage points (pp). The revised table reports all signed differences, both actual mean TTFTs, and their intervals. Seven positive estimates are not interpreted as seven statistically significant gains.

Descriptive comparisons of measured points. The following comparisons use evaluation results and do not replace the development-frozen comparisons. On Qwen2.5-VL/Video-MME, the highest-accuracy observed configuration of each baseline within ±2% of EWiM’s mean TTFT is:

MethodAccuracy (%)Mean TTFT (s)
EWiM61.02.16
VecAttention60.02.17
FlashVID + optimized exact attention59.62.15

The accuracy differences are 1.0 and 1.4 pp. Conditional on these configurations, source-video-clustered paired bootstrap intervals are [0.5, 1.5] and [0.8, 2.0] pp, respectively, from 10,000 resamples. These are nominal pointwise 95% intervals without adjustment for operating-point selection. The complete curves retain configurations outside this matching window.

For a separate post hoc quality band of at most 0.8 pp below Dense, applied to unrounded accuracies, the fastest qualifying measured VecAttention and FlashVID configurations achieve 60.6% at 2.39 s and 60.5% at 2.36 s. EWiM achieves 61.0% at 2.16 s: 9.6% and 8.5% lower mean TTFT, respectively. These comparisons describe the disclosed searches, not every possible configuration.

For this pair, EWiM accuracy averages five routing seeds (SD: 0.14 pp). Each bootstrap replicate recomputes the same five-seed mean; seeds are not treated as fivefold independent samples. Repeated-run timing uncertainty is reported separately from routing variability and request-population sampling.

Across the original development-selected operating points, arithmetic-mean accuracy retention is 99.2% and geometric-mean TTFT speedup is 1.46×, recomputed against the verified Dense reference. The added measurements characterize high-fidelity trade-offs within the reported searches, rather than uniform frontier dominance. VecAttention’s faster 58.6% / 1.65 s point remains in the Qwen2.5-VL/Video-MME plot. We report this speed–quality trade-off alongside the higher-fidelity comparisons in the revised Figure 4 and accompanying results tables.

AuthorsOpenReview ↗

Reply 2: ECA cost predictions, calibration, and reproducibility

Table 9 evaluates allocation outcomes; it does not independently validate cost predictions. We added separate validation of total refinement time, conditional marginal time, and actual layer-budget usage.

Independent validation. Calibration and validation trajectories are disjoint. Accepted sets AA and A∪{b}A\cup\{b\} are independently replayed from the same provisional baseline x0x^0, using the deployed batching and refresh schedule:

cmeas(b∣A;x0)=tref(A∪{b};x0)−tref(A;x0).c_{\mathrm{meas}}(b\mid A;x^0)=t_{\mathrm{ref}}(A\cup\{b\};x^0)-t_{\mathrm{ref}}(A;x^0).

Paired, interleaved repetitions control order effects and device drift. Fitting and validation use per-trajectory means. Data used to revise calibration are treated as development data and excluded from final independent validation.

Prediction errors. Across H800 deployment trajectories for the two primary 7B hosts, total refinement-time prediction has median/P90 absolute percentage errors (APE) of 4.2%/9.1%. On shapes excluded from fitting, these are 7.1%/13.8%. Results are also separated by host and shape.

For measured marginal costs ≥5 ms, median/P90 APE are 8.4%/17.6%. Below 5 ms, median/P90 absolute errors are 1.4/4.2 ms. These cases remain in the budget-boundary analysis: the 4.2 ms tail is not assumed negligible because it is expressed in absolute units.

Actual budget usage. Excess is assessed using the repeated-measurement mean for each held-out final ECA layer execution that invokes probing. This includes probes accepting no blocks; layers bypassed before probing are excluded and counted separately. Measured additional time exceeds BℓB_\ell in 1.3% of these layer executions. Conditional on an excess, its P95 magnitude is 2.2% of BℓB_\ell. Counts are reported per shape. This is a statistic of layer-execution means, not a per-request or single-run deadline guarantee.

The budget includes candidate construction, readout preparation, probing, selection, exact refinement, and refresh. Probing remains charged when no block is accepted.

Separately, ECA achieves 61.0% at 2.16 s mean TTFT, versus Static Cost’s 60.6% at 2.15 s. Their paired difference is 0.4 pp, with a source-video-clustered 95% CI of [0.1, 0.7], using the same routing-seed aggregation. Per-request means are 84 accepted blocks/34 query tiles for ECA and 78/44 for Static Cost. These counts describe allocation concentration; they do not establish prediction accuracy or attribute the accuracy gain exclusively to reuse. Complete-system TTFT includes all execution overhead.

Calibration and hardware transfer. Calibration uses 220 replay configurations per host–backend–GPU combination, without answer labels. The primary H800 calibration takes 41 minutes, including preparation, compilation, warm-up, and repetitions. The calibration table identifies this combination and reports the others separately.

On A100-80GB, we recalibrate cost coefficients and recompute budgets while retaining k0k_0, λB\lambda_B, probing layers, scoring, and candidate settings, without evaluation-based operating-point reselection. Total-time median APE is 7.4%; budget excess occurs in 3.8% of final ECA layer executions under the same definitions. Across the same eight main pairs, arithmetic-mean accuracy retention is 99.1% and geometric-mean TTFT speedup is 1.42×, relative to Dense on A100.

Exact configuration. The reference uses t=64t=64, k0=16k_0=16, Cℓ=256C_\ell=256, r=4r=4, a 50:50 ranked/coverage ratio, and microbatch size 32. Both primary hosts probe zero-based layers {12,14,16}\{12,14,16\}. With λB=0.25\lambda_B=0.25,

Bℓ=λBt^dense,ℓ,B_\ell=\lambda_B\widehat t_{\mathrm{dense},\ell},

where t^dense,ℓ\widehat t_{\mathrm{dense},\ell} is profiled Dense attention time for that layer and input shape. Budgets are milliseconds per layer; the revised overhead table pairs their sum with measured additional time for each shape.

Within each layer, ∣A∣|A| counts head-blocks and U(A)U(A) counts distinct valid query rows, deduplicated across heads. The coefficients αℓ\alpha_\ell, βℓ\beta_\ell, and γℓ\gamma_\ell have units ms/head-block, ms/valid row, and ms of startup cost for a nonempty set. Revised configuration tables specify every host’s layers, profile granularity, and counter mapping. The cost-validation, calibration, and overhead tables provide the corresponding measurements.

These are profile-based budgets, not hard wall-clock guarantees. Changes in hardware, kernels, precision, or execution schedule require recalibration and independent validation.

AuthorsOpenReview ↗

Reply 3: Repeated-query accuracy, latency, and reuse

Figure 8 evaluates MM independent questions about the same video, not an accumulated multi-turn conversation. We have clarified the reuse protocol and report answer quality alongside cumulative TTFT for each workload.

Actual reuse and consistency checks. The revised reuse table specifies cached objects, construction and invalidation conditions, and computation skipped on cache hits. EWiM recomputes question-conditioned scores, block selections, and dependent states for each question. Only verified question-independent states preceding question-dependent refinement are reused.

On a fixed 200-request Qwen2.5-VL/Video-MME diagnostic panel, we compare each deployed cache path with independent recomputation under identical decoding settings and routing seeds. EWiM and the compared methods using compatible reuse produce identical answers across the two paths on all 200 requests. This is an answer-level check, not a benchmark accuracy estimate or proof of universal internal-state equivalence.

Separately, for each M∈{1,3,8}M\in\{1,3,8\}, quality is evaluated on the complete evaluation set for that workload. Cached and independent executions use the same videos, questions, decoding settings, and routing seeds. Their absolute accuracy difference is at most 0.1 pp for each MM. These are complete-workload results, not estimates from the diagnostic panel. They show closely matching aggregate accuracy, not per-question answer identity or computational equivalence.

Quality–latency trade-off. The results table accompanying revised Figure 8 reports Dense, VecAttention, and EWiM accuracy and cumulative TTFT together for each workload. All methods use the same videos and questions within each MM; comparisons are made within workloads, not across different question sources. Cache construction, reads, transfers, and refresh are included in cumulative timing. Peak workspace is reported separately.

The limitation remains: at three and eight questions per video, EWiM’s cumulative TTFT exceeds Dense’s, while VecAttention remains faster. The reuse-consistency checks therefore do not establish an EWiM efficiency advantage in these settings. The accompanying accuracy results show the quality trade-offs at these measured operating points.

Our primary efficiency claim concerns single-request inference without cross-request visual-feature or KV reuse. This specifies the evaluation protocol, not an intrinsic inability to share question-independent states. TTFT begins with decoded frames in host memory and ends at first-token logits, including visual encoding and all acceleration overhead. We use T(M)/MT(M)/M only as amortized first-token latency, not throughput or end-to-end generation latency. We do not extrapolate the single-request speedup to cached multi-question workloads or infer an unmeasured query-count crossover.

Comments on the paper

2 comments
Area_Chair_ctyvOpenReview ↗

Reminder for discussions

Dear Reviewers,

The ICLR 2027 discussion phase will end on November 18 at 11:59 pm AoE. If you haven't already, please review the rebuttal and provide concrete, justified feedback to the authors on whether it addressed your concerns. Active participation is critical to the conference and the authors.

Thanks for your contribution to ICLR 2027!

-AC

AuthorsOpenReview ↗

Request for AC assessment of the revised manuscript and FlashVID results

Dear Area Chair,

We ask that you invite Reviewer zjhQ to assess the revised Section 3 during the reviewer discussion and give your own assessment of the remaining practical concern in the meta-review. In their November 18 follow-up, the reviewer said they understood our method clarification and asked us to incorporate it into the paper. We uploaded the revision after that comment. Section 3 now starts from standard masked attention, defines the notation together, and follows one refinement from its estimated effect on the question representation through selection and execution. The reviewer’s last public assessment therefore predates the version that addresses their writing concern.

On practical value, we agree that the near-equal-latency comparison with FlashVID covers one model and benchmark. The broader comparison in Table 1 covers two models and four benchmarks. Across all eight primary settings, EWiM gains 0.6–2.3 accuracy points over FlashVID at an additional 0.08–0.20 s in time to first token. On Qwen2.5-VL-7B with Video-MME, the latency sweep gives 61.0% at 2.16 s for EWiM and 59.6% at 2.15 s for FlashVID (Fig. 4e). A simpler alternative that spends the time on a denser single pass reaches 60.1% at 2.17 s. This last control directly tests whether the refinement step earns its computational cost.

The rebuttal also added independent validation of the execution-cost model on held-out trajectories (Table 12). We recalibrated on a second GPU and re-evaluated accuracy and latency across all eight settings with the routing parameters fixed. On A100, EWiM retained 99.1% of dense accuracy on average with a 1.42× geometric-mean speedup in time to first token (Table 13). In response to the request for a runtime breakdown, Table 14 reports profiles at 8K, 16K, 32K, and 64K tokens. We also report where the method loses its advantage: with three or eight questions on the same video, dense prefix reuse gives lower cumulative latency (Table 15).

We believe these results support acceptance. They establish that question-guided refinement can improve accuracy at nearly identical latency for a known question on a fresh video, and they quantify how that trade-off changes across models and workloads. The prevalence of this workload in deployment remains open. We ask that the final decision weigh that specific limitation against the demonstrated contribution and assess the writing on the revised manuscript.

Thank you for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.