acceptodds
Under review as a conference paper at ICLR 2027

Entropy Flow Graphs: Analytic Spatio-Temporal Graph Representations for No-Reference Video Quality Assessment

Abstract

Video quality failures have spatio-temporal structure, yet no-reference video quality assessment (NR-VQA) collapses every failure into a single scalar. We introduce the Entropy Flow Graph (EFG), a quality representation that fixes its structure analytically and confines learning to three matrices. Region nodes carry a five-cue entropy decomposition, spatial edges measure the conditional entropy deviation of a region’s co-statistics from natural regularities, and motion-compensated temporal edges let the aligned past of the video serve as its own reference, so the representation is simultaneously region-first, temporal, no-reference, and localizing, with edge weights in closed form. The analytic structure yields frame-rate invariance, permutation invariance, a dominance decomposition separating global from transient degradations, and plug-in consistency with an explicit bin-count rule. On the region-labeled D-CQA corpus EFG attains a global SRCC of 0.897 against 0.831 for the strongest holistic baseline with a localization AUC of 0.823, transfers zero-shot to KoNViD-1k and LIVE-VQC, and, serialized as structured context, lifts the region-level quality reasoning of a multimodal LLM by 12.6 points, more than a learned categorical graph under the identical protocol. We release splits, metrics, and code that make D-CQA a reusable benchmark for spatio-temporal quality localization.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
No-reference video quality assessment, Spatio-temporal graph representations, Conditional entropy, Spatio-temporal quality localization, Multimodal large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.80 over 5 reviews, above 85% of the others.
Reviewer_eFaoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper builds an entropy-based graph for no-reference video quality assessment. A small readout predicts quality, while spatial and motion-compensated temporal edges provide localization maps. It also introduces D-CQA, a CCTV dataset with local quality labels, masks, and onset annotations, and reports quality, localization, transfer, and multimodal-context results.

Critical strengths
  1. Interpretable localization signals. The metric produces interpretable localization maps: spatial responses reflect deviations from expected image statistics, while temporal responses reflect changes after motion compensation.

  2. D-CQA annotation design. D-CQA combines local quality ratings with degradation masks and onset annotations. This annotation design supports separate evaluation of perceived quality, spatial extent, and temporal onset.

Critical weaknesses
  1. The guarantees and localization ablations are inconsistent with the method. The proof of Proposition 4 treats CED in Eq. (5) as an average of the conditional entropies in Eq. (4), overlooking the absolute value. Taking the absolute value turns sampling errors around zero into positive contributions, so the bias can decrease more slowly than claimed. The refinement proof in Proposition 5 assumes that smaller regions have histograms close to that of their parent. As regions shrink, they can become nearly homogeneous while the parent still contains a mixture of different textures or intensities, so their histograms can remain far apart. There is also an experimental inconsistency: the localization map in §3.6 depends only on the graph, so changing readout depth alone cannot change its AUC on the same graphs and test data. Tables 2 and 4 nevertheless report such changes.

  2. The evaluation does not establish an advance over existing local VQA. Patch-VQ already provides human ratings of local video regions and predicts space-time quality maps, contradicting the broad firstness claim. Its omission leaves the comparison incomplete. EFG also receives local supervision, but comparable supervision for the baselines is not documented. The experiments therefore do not separate the value of the graph from that of additional labels. The localization metric takes the strongest spatial response across the clip and discards timing, so it cannot validate the claimed ability to locate degradation onset.

Reference Zhenqiang Ying et al. “Patch-VQ: ‘Patching Up’ the Video Quality Problem.” CVPR, 2021

Questions for authors
  1. Which theoretical guarantees remain valid after accounting for the bias and refinement issues, and how should the localization ablations be interpreted?
  2. What evidence isolates the contribution of the graph from local supervision and demonstrates temporal localization?
3 comments

In the revised paper, the authors have addressed several issues regarding the proofs and the ablation study in W1 / Q1. Furthermore, several new baselines (Patch-VQ, Q-Align, SimpleVQA, FasterVQA, regional-head controls, temporal metrics) have been added to the paper.

When re-reading the revised paper in detail, there still remains one major open question in W2, i.e., how EFG relates to general NR-VQA? First, the paper presents EFG as a general NR-VQA method, without even mentioning CCTV until §5.1, while D-CQA uses fixed surveillance views with synthetic corruptions that are naturally aligned with the inductive biases of EFG. For example, the static background in D-CQA reduces the temporal self-reference w_t to near background subtraction (whereas content motion triggers w_t on clean walkers in Fig. 8), the pasted local masks create sharp borders that are specifically designed for CED (w_s), and no whole-clip MOS is collected—the global target y is simply the 4×4 tile mean, which matches the area pooling of EFG even for lambda_r = 0. Second, the paper investigates the question of cross-dataset transfer without standard LSVQ training (see Table 1). Training general VQA backbones on static synthetic CCTV data seems to degrade them before UGC transfer: on LIVE-VQC, D-CQA-trained DOVER (0.766) and FAST-VQA (0.741) fall well below their standard LSVQ-trained results (0.832 and 0.823, respectively).

Can the authors report EFG trained under the standard LSVQ protocol (supported by Eq. 10 and LSVQ v-patches) on LSVQ_test, LSVQ_1080p, KoNViD-1k, and LIVE-VQC, together with the official LSVQ-trained baselines, and also clarify whether EFG is intended as a general NR-VQA method or as a method for fixed-camera monitoring?

Thank you for re-reading the revision and for the precise follow-up. We answer the scope question first, because it decides the rest. A further revised PDF is uploaded with the changes below marked in blue.

1. Scope. EFG is intended as a method for fixed camera monitoring. The evidence in the paper supports that scope and does not support a claim of general NR-VQA, and you are right that the submission did not say so before Sec. 5.1. The abstract, the introduction, and the conclusion now state the fixed camera scope, and Sec. 5.2 describes the UGC columns of Table 1 as a robustness check under domain shift, not as a general NR-VQA ranking. We kept the title unchanged during the discussion and will narrow it in the final version if the chairs permit.

2. Alignment between D-CQA and EFG. We agree that the benchmark favors the method. The three points are now stated as limitations in a new paragraph of App. B and are referenced from Sec. 6.

Your pointOur positionWhat bounds it
A static background reduces wtw_t to near background subtractionAgreed. Fixed views are the design premise of the temporal self referenceThe warp absorbs camera motion (Fig. 6, wtw_t from 0.269 to at most 0.038), but content motion still triggers wtw_t (Fig. 8), which we measure as 7.1% false positives on clean clips (Table 4)
Local masks with sharp borders favor CEDPartly agreed. The masks are feathered, so there is no step edge, but the transition band is narrow and the border stays fairly sharpThe regions come from the segmenter and not from the mask, and 40% of the clips carry a global degradation with no border at all
No whole clip MOS is collected and yy is the tile meanAgreed. This matches the area pooling of EFG even at λr=0\lambda_r = 0The external sets with whole clip MOS are the only evidence free of this alignment

3. Transfer. You are right that training general backbones on synthetic CCTV data handicaps them. Table 1 now lists the official LSVQ trained results as reference rows, and Sec. 5.2 states plainly that EFG does not match them.

ModelTraining dataKoNViD-1k SRCCLIVE-VQC SRCC
FAST-VQAD-CQA0.7840.741
DOVERD-CQA0.8010.766
EFG, λr=0\lambda_r = 0D-CQA0.8370.789
EFGD-CQA0.8420.795
FAST-VQA (official)LSVQ0.8590.823
DOVER (official)LSVQ0.8840.832

The transfer columns therefore answer a narrower question, namely what survives when the only training data is synthetic surveillance footage. Under that same training data EFG stays ahead of the general backbones, and we claim no more than that.

4. LSVQ protocol. This is the right experiment for a general claim, and EFG can be trained that way through Eq. (10), with the LSVQ space time patch labels as the region set. Building the graphs for the LSVQ videos does not finish before the discussion closes, so we cannot report it now and prefer not to report partial numbers. With the claim narrowed to fixed camera monitoring the paper no longer rests on it, and App. B names the LSVQ protocol as the test of whether the representation extends beyond fixed views.

Thank you for providing the detailed feedback on my questions. I increased my rating.

Reviewer_seknOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

It is a well written paper on entropy flow graphs (EFGs) in no-reference VQA. The proposed work introduces a theoretically grounded visual quality representation, which addresses two major gaps - spatio-temporal coarseness of current models and the lack of fine-grained subjective/human judgements below frame-level. EFG models a video as a spatiotemporal graph, and limits learning to only three lightweight readout matrices making the graph edges themselves directly legible as temporal and spatial localization maps. The authors collect and release D-CQA, the first spatiotemporal quality database with 60,000 video clips with region-level subjective scores, which elevates the relevance of the contribution.

Critical strengths
  1. The proposed spatiotemporal edges have closed-form mathematical representations for fundamental properties like frame rate and permutation invariances, dominance decomposition, etc.
  2. A dataset containing 60,000 annotated clips with region-wise human labels and traced degradation extents represents a solid contribution
  3. EFG's zero-shot generalization provides validation about NSS priors being robust.
Critical weaknesses
  1. The model seems to struggle with texture masking (as shown in the qualitative results, figure 6)—where dense, busy textures mask local degradations. Because the entropy of busy natural textures is already high, local noise or blur does not alter the entropy distribution enough to stand out, which is a known fundamental limitation of entropy-based metrics.
  2. The motion-compensated temporal edges rely heavily on a two-stage warping scheme (translation phase correlation followed by optical flow). In scenes with complex occlusions, specular reflections, or rapid illumination changes, optical flow estimation can fail, leading to spurious "hot" temporal edges that represent registration errors rather than visual degradations.
  3. EFG is dependent on an upstream segmenter (ex. SAM). If the segmenter produces inconsistent region boundaries across frames or is too computationally heavy, the overall speed of the pipeline degrades. (The grid-based fallback that the authors suggest comes with a minor accuracy cost.
Questions for authors
  1. Have you considered integrating a local optical flow confidence map (e.g., checking warp consistency or forward-backward error) to down-weight temporal edges in regions where registration is known to fail?
  2. When using SAM masks, the segmenter accounts for the vast majority of the per-frame runtime (74 ms at 13 fps). Are there lighter, real-time segmenters (like MobileSAM) that can match SAM’s accuracy without the latency penalty?
  3. Currently, temporal edges only connect adjacent resampled frames (t to t+1). If a video suffers from slow, drifting degradations (like gradual defocusing over 3 seconds), does the localized step-wise conditional entropy fail to capture it?
Reviewer_asrrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes an entropy-based spatiotemporal graph for no-reference video quality assessment, using spatial and motion-compensated temporal edges to produce regional, frame-level, and global quality scores. It also introduces D-CQA with 60,000 region-annotated clips and reports improvements in quality prediction, localization, cross-dataset transfer, and multimodal-LLM-assisted distortion identification.

Critical strengths
  1. Explicit localization with multi-level quality outputs. Using graph edges directly for localization is an appealing design that connects quality prediction with interpretable spatial and temporal evidence.

  2. Valuable regional annotations and empirical results. D-CQA provides valuable regional supervision, and the reported gains over DOVER: 0.897 versus 0.831 global SRCC and improvements on three transfer datasets provide encouraging empirical evidence.

Critical weaknesses
  1. The finite-sample bias bound in Proposition 4 is not established. Appendix C.4 turns an asymptotic bias expansion into a finite-sample upper bound without controlling its remainder, and does not propagate estimation error through the absolute deviation in Eq. (5), leaving Proposition 4 and the theoretical justification for the bin-count rule insufficiently supported.

  2. The key step in the proof of Proposition 5 is unjustified. The refinement proof incorrectly infers that a small child-region diameter makes its histogram close to the parent histogram; a small, nearly homogeneous child can differ substantially from a heterogeneous parent, so Eq. (14) does not follow from the stated assumptions.

Questions for authors
  1. Are the splits disjoint by camera/location as well as source clip, and how was repeated scenery across different source clips excluded?

  2. Is the global target y an independently collected whole-video judgment or an aggregation of tile ratings, and what aggregation and normalization are used?

Reviewer_ZbiBOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces the Entropy Flow Graph (EFG), which formulates no-reference video quality assessment as inference over a region-level spatiotemporal graph. Nodes encode the entropies of five visual cues, spatial edges measure conditional entropy deviations between adjacent regions, and temporal edges capture motion-compensated conditional entropy. A lightweight graph network produces region-, frame-, and video-level quality scores, while the analytic edge weights also serve as degradation localization signals. The authors additionally introduce D-CQA, a dataset containing 60,000 synthetically distorted videos with region-level quality scores and degradation extent annotations. EFG reportedly achieves a global SRCC of 0.897, a region-level SRCC of 0.764, and a localization AUC of 0.823 on D-CQA. The paper also demonstrates cross-dataset transfer for global quality prediction and benefits from providing the graph as structured context to multimodal models. The problem is well motivated and the approach has potential practical value. However, inconsistencies in the localization evaluation and gaps in some theoretical arguments prevent the current evidence from fully supporting the claimed contributions.

Critical strengths
  1. The paper proposes a practically meaningful framework for region-level spatiotemporal quality assessment. Extending global quality prediction to region-level scores and spatiotemporal degradation evidence provides more useful diagnostic information. Spatial conditional entropy deviation and motion-compensated temporal conditional entropy have explicit statistical interpretations, and the quality prediction ablations support their complementary roles. For example, removing temporal edges reduces global SRCC by 0.171, while removing spatial edges reduces region-level SRCC by 0.187. This gives the approach methodological value beyond improvements in prediction accuracy alone.
  2. The region-level benchmark is a potentially valuable contribution in its own right, and the reported quality prediction results are competitive. D-CQA provides supervision and evaluation targets for local video quality beyond a single global score. EFG improves global SRCC on this dataset from DOVER’s 0.831 to 0.897 and maintains an advantage on three external datasets without retraining. These results motivate further investigation of the representation, although complete dataset documentation and fair controlled comparisons remain necessary to establish the sources of the gains.
  3. The theoretical analysis informs concrete design choices rather than serving a purely descriptive role. Proposition 1 motivates anchoring temporal sampling to a physical interval (\Delta) rather than frame indices; the comparison at 15 fps, with score drift of 0.006 versus 0.081, provides a controlled test of this choice. Proposition 4 yields an actionable rule for selecting the histogram bin count (B), with supporting evidence in Fig. 7(c). The segmentation refinement plateau predicted by Proposition 5 is observed beyond approximately 64 regions per frame. These propositions connect directly to specific design decisions, although the validity of some guarantees remains a concern, as discussed below.
Critical weaknesses
  1. The localization evaluation is inconsistent with its stated definition, undermining the central localization claim. Section 3.6 defines the spatial localization map (M_s) as a mapping of the analytic spatial edge weights, and Section 5.1 explicitly evaluates localization AUC using (M_s) against the annotated degradation extents. Under this definition, changing only the depth of the downstream graph network should not alter (M_s). However, Table 4 reports that reducing the readout depth to one layer decreases localization AUC by 0.011, while increasing it to three layers improves AUC by 0.002. Removing temporal edges also decreases AUC by 0.202. The authors need to specify the localization map actually evaluated and its computational dependencies. If the evaluated map incorporates network predictions or spatiotemporal fusion, the claim that analytic edge weights directly provide localization without additional learning, along with the interpretation of the ablations, requires revision. Furthermore, temporal localization is primarily illustrated through examples and lacks a separate quantitative evaluation.
  2. The current proofs do not establish all of the claimed theoretical guarantees, particularly Proposition 5. Appendix C.5 infers that a refined region’s histogram is close to its parent region’s histogram because the refined region has a small diameter. This implication does not generally hold. For example, a large region with smoothly varying luminance can have a broad histogram, while its small subregions each have concentrated histograms. The child and parent distributions can therefore differ substantially even when variation within each child is small. Consequently, the current proof does not establish the claimed segmentation refinement stability. The locality argument in Proposition 3 also needs to clarify whether segmentation and motion correspondences are held fixed, since a local distortion may change the re-estimated optical flow or segmentation. Empirical stability trends cannot substitute for these missing theoretical arguments, especially when analytic structure and its guarantees are emphasized as central contributions.
  3. The baseline selection and adaptations do not adequately isolate the claimed advantages. Q-Align, SimpleVQA, and FasterVQA appear in the parameter–accuracy plot in Fig. 10(a), but none appears in the main comparison in Table 1. Q-Align is a representative LMM-based quality assessment method and could be a strong competitor after fine-tuning on D-CQA; its inclusion in the figure but exclusion from the main table requires explanation. The graph-based comparisons are also weakened by their adaptations: PANDA uses the first frame as its anchor, while GraphIQA is applied frame by frame and then pooled. Both originate from static image assessment, and these adaptations may substantially disadvantage them in video evaluation. PANDA’s global and region-level SRCC values of 0.694 and 0.518 therefore do not provide a sufficiently controlled basis for concluding that analytic structure is superior to learned structure. In addition, comparing EFG’s 0.08M trainable parameters with DOVER’s tens of millions does not disentangle representation quality, model capacity, and the strength of the imposed inductive bias. Capacity-matched controls and experiments providing the same entropy-graph information to an alternative backbone, such as DOVER, are missing.
Questions for authors
  1. Localization metric clarification: Is Loc. AUC computed from the raw spatial edge map, predicted region-level quality scores, or a fused spatiotemporal map? Please provide the complete computation and explain why changing only the readout depth changes localization AUC. If the implementation differs from the definitions in Sections 3.6 and 5.1, what are the corrected main results and ablations?
  2. Theoretical correction: What additional assumptions are required to establish that a refined region’s histogram is close to its parent region’s histogram in Proposition 5? Can the authors provide a valid proof? Does Proposition 3 apply only when segmentation and motion correspondences are fixed, rather than when the entire graph construction pipeline is rerun?
  3. Dataset split clarification: Are the D-CQA training, validation, and test sets disjoint at the level of the 3,000 source videos, or are they randomly partitioned at the level of the 60,000 distorted clips? Please state the split procedure explicitly and report SRCC under a source-disjoint split.
  4. Missing baselines: Why are Q-Align, SimpleVQA, and FasterVQA absent from Table 1 despite appearing in Fig. 10(a)? Please report their SRCC and PLCC after fine-tuning on D-CQA under the same protocol, and clarify whether each method can produce region-level quality scores.
Reviewer_vz4AOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes the Entropy Flow Graph (EFG) for no-reference VQA.

Each resampled frame is partitioned into regions (SAM masks snapped to a 16×16 grid).

Spatial edges are weighted by the deviation of boundary conditional entropies from a regressor fitted on clean images (CED), and temporal edges are weighted by the conditional entropy of a region given its motion-compensated past.

The paper states five propositions (frame-rate and permutation invariance, a spatial/temporal dominance decomposition, plug-in consistency with a bin-count rule, and refinement stability).

And introduces D-CQA, 60,000 synthetically degraded CCTV clips with tile-level ratings and traced degradation extents.

Critical strengths
  1. They propose an interpretable, no-reference design for localized VQA. And address a real gap: NR-VQA models give no indication of where or when quality fails, and EFG addresses this with a very small learned component.

  2. Tile-level video ratings with traced extents and onsets, together with per-cue and per-type ablations, a sensitivity study, transfer tests, and explicit failure cases, would support future work on localized VQA.

Critical weaknesses

Firstly, Sec. 3.6 defines the spatial map MsM_s by rasterizing the analytic weights wsw_s with "no threshold and no extra parameters", and Sec. 5.1 defines Loc. AUC as the ROC AUC of MsM_s against the traced extents.

Under this definition, Loc. AUC cannot depend on the learned readout, yet Table 4 reports changes for readout depth (L = 1: −0.011; L = 3: +0.002).

Table 4 also reports a drop of −0.202 when temporal edges are removed, although MsM_s does not use them, and a finite value (−0.174) when spatial edges are removed, in which case MsM_s does not exist.

An analytic map has no seed dependence, yet the results are averaged over three seeds. The reported localization therefore appears to come from learned region scores or a combined map.

In addition, "when" is never evaluated quantitatively: Loc. AUC uses only MsM_s, MtM_t appears only in Fig. 6, and false positives caused by content motion (App. G) are never measured.

Since parameter-free "where and when" localization is the paper's main distinction from holistic NR-VQA, this weakness weighs most in my recommendation.

Secondly, each headline contribution is overstated as currently supported.

(a) Accuracy gain: EFG is trained with a region loss (Eq. 10, λr=0.3\lambda_r = 0.3) whose labels average to the global target, while the baselines see only the global label, and λr=0\lambda_r = 0 is not included in the sweep (Table 5). The 0.066 SRCC gap over DOVER therefore mixes architecture with supervision granularity.

(b) Benchmark: Patch-VQ (Ying et al., CVPR 2021) already provides human ratings for 116,433 space-time localized video patches, together with a space-time quality mapper. The claim that no video benchmark offers sub-frame human judgments is therefore incorrect, and Patch-VQ is not cited.

The stated protocol also implies at least ≈48M tile ratings from 24 raters, and the source of the CCTV footage is unclear.

(c) Analysis: The proof of Prop. 5 fails: a small child diameter does not make child histograms close to the parent mixture.

A luminance ramp spanning all B bins has parent entropy log⁡B\log B, while children that each fall in a single bin have entropy 0.

Eq. (11) does not hold for CED because of the absolute value in Eq. (5): when the true deviation is zero, E∣H^n−H∣E|\hat H_n - H| scales as n−1/2n^{-1/2}.

The Miller–Madow expansion is also used as a finite-sample bound, and the bin-count arithmetic does not match the stated geometry: a one-cell boundary gives ≈203 pixel pairs, so the paper's own rule yields B ≤ 7, not B = 8.

[1] Ying, Zhenqiang, et al. "Patch-VQ:‘Patching up’the video quality problem." 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2021.

Questions for authors
  1. How exactly is Loc. AUC computed (pixel or tile level, per clip or pooled, and from which map)? Why does it change with LL, with EtE_t removed, and with EsE_s removed? Please report the AUC of the analytic MsM_s and MtM_t alone.

  2. What is EFG's global SRCC with λr=0\lambda_r = 0, and what is the SRCC of at least one baseline trained with the same region supervision (e.g., a region head on a FAST-VQA/DOVER backbone)?

  3. How many tile ratings were actually collected, over how many hours, and were all tiles of all sampled frames rated? Which source corpora provided the CCTV footage, how many distinct cameras are there, are the splits camera-disjoint, and do any sources overlap with CQA-DB?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.