acceptodds
Withdrawn from ICLR 2027

Few-Query Quality Assessment for Unseen Machine Observers: Gauge-Invariant Decision Subspaces

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Machine-oriented image quality is observer-dependent: visual models can prefer different processed versions of the same image. We study few-query calibration on a fixed content–candidate bank: source models provide complete responses, while a previously unseen observer reveals only a few candidate groups before its choices on other groups are predicted. A reconstruction-optimal low-rank basis can discard decisive candidate comparisons, and an empirical-Bayes posterior can depend on arbitrary factor coordinates unless its prior transforms with the basis. We introduce Gauge-Invariant Decision Subspaces (GIDS), which selects a decision-aligned subspace within a spectral dictionary and estimates its observer prior and residual noise in matching coordinates. Training simulates complete-group calibration; deployment uses a closed-form low-dimensional posterior. Theory establishes gauge-covariant prediction and a regret bound separating representation from posterior-identification error. Against a matched gauge-consistent SVD baseline, GIDS lowers observer-macro selection regret from 0.1282 to 0.1266 under CIFAR-10 architecture-family holdout, with mean improvements on prospectively frozen observers and a frozen CIFAR-100 domain. The results show the value of choosing a low-rank representation for decisions after calibration, rather than response reconstruction alone.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
machine-oriented image quality assessment, unseen machine observers, few-query calibration, gauge invariance, decision-focused learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_r59WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Based on my understanding, the paper aims to address the following problem: Given the evaluations of candidate images from several source models, can we predict a new model's preferences over the remaining candidate images by querying only a small number of its responses, and thereby select the version with the highest utility?

I believe the paper adopts a well-motivated optimization objective, and the key experiments are sufficiently comprehensive and fair. The mathematical derivations are also relatively rigorous. However, I am not fully convinced that the reported improvements are stable and consistent across different settings. Furthermore, I believe that the main contribution lies in the task-specific integration and design of existing techniques, while the novelty of the core technical components themselves is relatively limited.

Critical strengths

(1) The paper highlights an important distinction: accurately predicting the numerical values of all responses is not necessarily equivalent to correctly selecting the optimal candidate version.

(2) The baseline used in this paper, SVD + empirical Bayes (SVD+EB), is well chosen and appropriate for the task.

(3) The paper reports extensive experimental results, including content-level, model-level, and model-family-level bootstrap analyses, providing a relatively comprehensive evaluation of the results.

Critical weaknesses

(1) The evidence is still insufficient to demonstrate whether the proposed method can consistently benefit observers from previously unseen model families.

(2) The main contribution lies in the task-specific integration and design of existing techniques, while the novelty of the core technical components themselves is relatively limited.

(3) The applicability of the proposed method is relatively limited, and the cost of constructing and maintaining the source response library requires further justification.

Questions for authors

(1) Under what conditions is the episodic approach more effective?

(2) Why does dense-fit outperform the proposed method in the main experiments?

2 comments
AuthorsOpenReview ↗

Response to r59w

We sincerely thank Reviewer r59W for the thoughtful review and constructive questions. We address the two related questions together below.

Response to Question 1 & Question 2 (when episodic training is more effective and why dense fit is stronger on CIFAR-10)

Both variants learn a decision subspace within the same spectral dictionary. Dense fit uses complete source-response coordinates, while episodic training optimizes decisions after few-query calibration.

Evaluation conditionGIDS (dense fit) ↓GIDS (episodic) ↓Paired Δ_epi−dense ↓Lower mean regret
CIFAR-10, all calibration groups0.11910.1199+0.0008Dense fit
CIFAR-10 development holdout, all budgets0.125800.12664+0.00084Dense fit
Five frozen confirmation observers, K ≥ 320.072950.07165−0.00129Episodic
CIFAR-100 transfer, K ≥ 320.110560.11030−0.00026Episodic

Why dense fit is stronger on CIFAR-10. Training with complete source coordinates provides a more direct decision-fitting signal, consistent with dense fit’s lower development regret. Both variants nevertheless outperform SVD+EB (0.1282) in the main comparison, supporting the value of decision-oriented subspaces.

When episodic training helps. Episodic training matches the few-query adaptation used at deployment. It achieves lower mean regret in both evaluated frozen-transfer settings, improving over dense fit by 0.00129 across five confirmation observers and 0.00026 on CIFAR-100. Thus, dense fit’s development advantage does not extend to these transfer means, supporting episodic training’s additional transfer value.

Reviewer_r59WOpenReview ↗

Keep my score

Thanks for the authors responce. I agree that some explaination is helpful, like the comparison between dense fit and episodic training. However, my main concern is if the proposed method can improve the performance for unseen model??

Please calrify if the these 5 comfirmantion observers belong to model families excluded from source-library construction, training, and model selection??

Thanks

Reviewer_hMyHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

GIDS learns to choose processed image candidates for an unseen classification model using only a few queries to that model. It assumes a fixed image-candidate bank with dense responses from source models, and each query reveals a complete candidate group. The method learns a rank-three decision subspace inside an SVD dictionary, then fits the target model with a Gaussian empirical-Bayes posterior matched to that subspace. The paper gives a coordinate-invariance result and a conditional margin-based regret bound. Against matched reconstruction-based and decision-aware controls, GIDS lowers CIFAR-10 observer-macro normalized regret from 0.1282 to 0.1266; the frozen-observer and CIFAR-100 results show smaller or more variable gains.

Critical strengths
  1. The SVD + EB comparison isolates the representation choice well: the methods use the same posterior solver and query groups, and the objective swap stays within the same spectral dictionary. The unrestricted decision-aware alternatives, posterior ablations, and query-policy controls help explain where the gain comes from. The 1.14-percentage-point increase in choosing the true utility maximizer also shows that the method changes candidate choices.

  2. The paper is clear about what information is available. It states the fixed-bank assumption, the label requirement, and the cost of building the source response bank. It also reports the uneven observer effects and the intervals over observers and families, and acknowledges that the independence assumption in the search bound does not hold for the experimental training procedure.

Critical weaknesses
  1. The average regret reduction is 0.00159, with observer and family bootstrap intervals of [−0.00441, 0.00059] and [−0.00378, 0.00093]. The two prospective targets tell rather different stories: SODEF changes by −0.00004 and WRN-A4 by −0.00296. On CIFAR-100 the family-macro difference is about −0.0004, also with an interval crossing zero. The positive means are useful evidence on these collections, but the variation leaves considerable uncertainty about the benefit for a future observer. That matters here because adaptation to a new observer is the main use case.

  2. I also find the methodological advance fairly modest. Transforming a Gaussian prior along with its basis is a basic consistency requirement, already satisfied by SVD + EB. The ablations attribute nearly all of the gain to the decision subspace; recomputing covariance and noise, and adding the projector anchor, have little effect. Dense decision fitting within the same dictionary does better on CIFAR-10 than the primary episodic method (0.1258 versus 0.1266). The frozen-transfer differences offer limited evidence for preferring episodic calibration. Taken together, the incremental method and modest, uneven confirmation results leave me just below the acceptance threshold, despite the careful experiments.

Questions for authors
  1. How much does the mean gain change when each architecture family is excluded from the aggregate in turn? Please give the per-family paired regret differences as well. The leave-one-observer-out result is helpful, but does not answer whether the advantage survives removing the most favorable family.

  2. For the frozen observers and CIFAR-100, please report the directly paired episodic-minus-dense-fit differences, with uncertainty over observers or families at the same evaluation endpoint. This would help establish whether differentiating through calibration adds a benefit beyond dense decision fitting.

  3. What is the paired change in classifier correctness between GIDS and SVD + EB? Table 8 gives correctness for GIDS, whereas Table 17 measures agreement with the negative-cross-entropy maximizer. A comparison of actual correctness would help judge the practical value of the regret reduction.

3 comments
AuthorsOpenReview ↗

Response to Reviewer hMyH, Q1

We sincerely thank Reviewer hMyH for the careful reading and constructive feedback. We address all concerns below.

Response to Question 1 (leave-one-family-out sensitivity and per-family paired regret differences)

In response to your request, we computed the per-family paired regret differences (GIDS minus SVD + EB) across all six architecture families on CIFAR-10, alongside the leave-one-family-out (LOFO) aggregate differences when each family is excluded in turn.

Architecture Family (# Models)Per-Family SVD + EB ↓\downarrowPer-Family GIDS ↓\downarrowPer-Family Paired Δ↓\Delta \downarrowLOFO Observer-Macro Δ\Delta (Excl. Row Family) ↓\downarrowLOFO Family-Macro Δ\Delta (Excl. Row Family) ↓\downarrow
WideResNet (3)0.1167850.109791-0.006994+0.000214+0.000021
XCiT (1)0.0618790.058953-0.002926-0.001467-0.000792
MobileNet (2)0.2309830.230480-0.000504-0.001805-0.001277
ResNet (3)0.1241450.124663+0.000518-0.002290-0.001481
ResNeXt (1)0.1063630.108096+0.001733-0.001890-0.001724
VGG (2)0.0928780.094163+0.001285-0.002163-0.001634
All 6 Families (12)0.128230.12664-0.00159—-0.00115 (Full Family-Macro)

The largest gains come from WideResNet and XCiT. The largest family-level reduction is 0.006994, about four times the largest increase of 0.001733. GIDS also retains lower average regret in five of the six LOFO cases under both weighting schemes.

AuthorsOpenReview ↗

Response to Reviewer hMyH, Q2

Response to Question 2 (directly paired episodic-minus-dense-fit differences on frozen observers and CIFAR-100)

We report directly paired regret differences, Δ_epi−dense = GIDS (episodic) − GIDS (dense fit), at the matched K ≥ 32 endpoint for all five frozen confirmation observers and CIFAR-100. Negative differences favor episodic training.

Frozen evaluation (K ≥ 32)GIDS (dense fit) ↓GIDS (episodic) ↓Paired Δ_epi−dense ↓Content 95% CIObserver / Family 95% CI
Five confirmation observers0.072950.07165−0.00129[−0.00220, −0.00039][−0.00480, +0.00145]
CIFAR-100 (7 families)0.110560.11030−0.00026[−0.00064, +0.00012][−0.00123, +0.00051]

Episodic training achieves lower mean regret in both frozen transfer evaluations, reversing dense fitting’s development-domain advantage. For the five-observer collection, the content-level interval excludes zero. These results support the transfer value of training through few-query calibration.

AuthorsOpenReview ↗

Response to Reviewer hMyH, Q3

Response to Question 3 (actual classifier correctness)

To directly evaluate the practical downstream impact on top-1 image classification accuracy ("Selected correctness" in Table 8, as opposed to negative-cross-entropy utility agreement in Table 17), we report the paired comparison of actual classifier correctness between SVD + EB and GIDS across all query budgets, using the same evaluation protocol as Table 8:

Query budget KSVD + EB accuracy (%)GIDS accuracy (%)Paired Δ (pp)
887.725087.7979+0.0729
1687.993887.9972+0.0035
3288.087588.0986+0.0111
6488.075088.0826+0.0076
12888.125088.1653+0.0403
Mean88.001388.0283+0.0271

Together with the 1.14-percentage-point increase in selecting the true utility maximizer reported in Table 17, these paired point estimates show that GIDS improves utility-maximizer agreement and achieves higher mean actual classifier correctness at all five query budgets.

Reviewer_ebf3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Gauge-Invariant Decision Subspaces (GIDS) for few-query, model-dependent image quality assessment. Given complete responses from existing machine observers, the method aims to predict the preferred image-processing candidates of an unseen observer using only a small number of queries. GIDS learns a decision-oriented low-rank subspace and employs gauge-consistent empirical-Bayes calibration. Experiments on CIFAR-10 and CIFAR-100 evaluate candidate selection under architecture-family holdout and limited query budgets.

Critical strengths
  1. The paper addresses the practical challenge of adapting image-processing preferences to previously unseen machine observers with limited evaluation queries.
  2. Optimizing the representation for candidate selection rather than response reconstruction is conceptually reasonable. The gauge-consistency analysis also provides a clear mathematical justification for the calibration procedure.
  3. The paper includes matched comparisons with SVD-based baselines, component ablations, different query budgets, and evaluations on held-out observer families.
Critical weaknesses
  1. The framework assumes a fixed candidate bank with complete source-observer responses. It cannot directly generalize to newly arriving images or processing operations, and target queries require ground-truth labels to compute task utilities.

  2. GIDS reduces average selection regret from 0.1282 to 0.1266 compared with SVD + EB. Moreover, observer- and architecture-family-level confidence intervals cross zero, raising concerns about the consistency and practical significance of the improvements.

  3. Experiments primarily focus on CIFAR-based classification tasks, with limited evidence for broader vision applications. Although the matched SVD baseline is appropriate, comparisons with stronger decision-focused, ranking-based, or simpler preprocessing-selection approaches remain insufficient.

Questions for authors
  1. Could the authors clarify the practical deployment scenarios in which a fixed candidate bank with complete source-observer responses is realistic? How would GIDS handle newly arriving images or changing processing candidates?
  2. Given the relatively small regret reductions and confidence intervals crossing zero at the observer and architecture-family levels, could the authors provide stronger evidence of consistent improvements across unseen models and more diverse tasks?
  3. Could the authors further clarify the methodological novelty relative to existing decision-focused learning and cold-start matrix factorization approaches, and compare against simpler model-specific preprocessing selection strategies?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ebf3, Q1

We sincerely thank Reviewer ebf3 for the careful reading and constructive feedback. We address all concerns below.

Response to Question 1 (practical deployment scenarios and handling new images or candidates)

In response to your valuable questions, we would like to clarify that the fixed-candidate-bank assumption is realistic in practical deployment scenarios where candidate outputs and their responses from source observers can be precomputed and reused. Furthermore, GIDS supports dynamic updates. Newly arriving images or candidate operations can be incorporated without additional target-observer queries, either through closed-form folding-in or inductive feature regression. We provide two representative deployment scenarios below:

  1. Hardware ISP / restoration preset calibration: Camera and autonomous-driving vendors maintain a fixed "golden calibration suite" (thousands of benchmark scenes pre-rendered under AA candidate ISP/denoising/tone-mapping presets) with precomputed responses from standard reference models. When a downstream team deploys a newly trained, quantized, or rate-limited proprietary model checkpoint, exhaustive hardware-in-the-loop testing over all NN candidates is prohibitive. GIDS calibrates per-scene preset selection for each new checkpoint from only 4–164\text{--}16 queried scenes.

  2. Cloud media repositories with pre-transcoded renditions: Media platforms store multiple pre-generated renditions (varying JPEG qualities, downsampling scales, or filters) per asset. When routing stored assets to a new pay-per-call vision API, GIDS identifies the API-optimal rendition per asset after a few probe groups.

To handle newly arriving images or changing candidates, GIDS decouples the target observer coordinate z^m∗\hat{z}_{m_*} (d=3d=3) from item rows bx,a⊤b_{x,a}^\top, allowing dynamic updates without re-querying calibrated target models:

Update ScenarioHow Item Coordinate bnew⊤b_{\text{new}}^\top is ObtainedNew Target Queries
New images / new candidate presets (Offline batch)Closed-form folding-in from source responses rsrc, new∈RMr_{\text{src, new}} \in \mathbb{R}^M via bnew⊤=rsrc, new⊤AQ(AQ⊤AQ)−1b_{\text{new}}^\top = r_{\text{src, new}}^\top A_Q (A_Q^\top A_Q)^{-1} (AQ=RBQA_Q = RB_Q)0
Streaming unseen images (Real-time)Predicted by an inductive feature regressor b^new⊤=ϕ(x,a)⊤WQ\hat{b}_{\text{new}}^\top = \phi(x,a)^\top W_Q trained on BQB_Q0
AuthorsOpenReview ↗

Response to Reviewer ebf3, Q2

Response to Question 2 (consistent improvements across unseen models and diverse tasks)

Beyond the overall mean regret, the appendix provides complementary evidence of improved candidate selection on the evaluated unseen observers. GIDS yields 8,178 net selection-error repairs (Table 12) and improves utility-maximizer selection accuracy by 1.14 percentage points (Table 17). The mean regret difference also remains negative after removing any single observer, ranging from −0.00192 to −0.00063 (Appendix C).

The numerical comparisons below summarize the original held-out-observer and transfer evaluations. On the top-quartile utility-margin subset at K ≥ 32, the regret reduction is −0.00517, approximately 2.30× the all-content reduction at the same budgets (−0.00225). This shows that the benefit extends to decisions with more clearly separated candidate utilities.

Evaluation settingSVD + EB ↓GIDS ↓Paired Δ ↓
CIFAR-10 unseen observers, all budgets0.12820.1266−0.00159
CIFAR-10 high-margin subset, K ≥ 320.23370.2285−0.00517
Prospective frozen targets, K ≥ 320.07990.0784−0.00150
CIFAR-100 unseen observers, K ≥ 320.11070.1103−0.00035

The kernel-prior extension (Appendix B) improves 6/7 held-out families and 59/70 family–direction–budget cells on CIFAR-100:

CIFAR-100, all budgetsGaussian ↓Kernel ↓Δ ↓
Fixed dense-fit subspaces0.11400.1078−0.00620

These results provide complementary decision and transfer evidence alongside the reported cluster-level uncertainty.

AuthorsOpenReview ↗

Response to Reviewer ebf3, Q3

Response to Question 3 (methodological novelty and comparison with simpler model-specific selection strategies)

In response to your valuable comments, we clarify that GIDS differs from cold-start matrix factorization (MF) and standard decision-focused learning (DFL) by optimizing a spectral-constrained Grassmann subspace through a gauge-covariant empirical-Bayes posterior. Furthermore, GIDS consistently outperforms both unrestricted DFL variants and simpler model-specific preprocessing selection heuristics. We detail the methodological distinctions below:

  1. Beyond Cold-Start MF (Gauge-consistent post-calibration optimization): Classical MF (e.g., PMF, SVD + EB) selects basis directions by global Frobenius energy and fixes the prior before calibration. GIDS proves that changing factor coordinates without co-transforming the prior alters posterior shrinkage (Proposition 1), and instead optimizes the projector QQ⊤∈Gr(r,d)QQ^\top \in \text{Gr}(r,d) directly through the closed-form posterior while dynamically re-estimating (CQ,σQ2)(C_Q, \sigma_Q^2) in each basis.

  2. Beyond Standard DFL (Spectral dictionary prevents cold-start overfitting): Standard DFL optimizes task loss in an unrestricted parameter space without a few-query calibration loop. In our controlled ablation (Appendix C, Table 11), unrestricted DFL (optimizing listwise or episodic loss across all item coordinates without our spectral dictionary FF) severely overfits source observers, degrading regret from 0.1266 to 0.1379 and 0.1338 (worse than SVD + EB at 0.1282).

Regarding the comparison against simpler model-specific preprocessing selection strategies, we evaluate GIDS under the identical CIFAR-10 protocol against published adaptations (Table 9) and two model-specific heuristics—selecting the target's best single operation on queried groups (Target Best Fixed Op) and copying the most correlated source observer on queried groups (Nearest Source 1-NN):

Selection StrategyMean normalized regret ↓
Target Best Fixed Op (new)0.2263
Nearest Source, Pearson 1-NN (new)0.1415
Unrestricted Listwise DFL0.1379
Unrestricted Episodic DFL0.1338
PMF (Cold-start MAP)0.1305
SVD + EB (Control)0.1282
GIDS (Ours)0.1266

Results are averaged over 12 held-out target observers, two transfer directions, and five query budgets (K ∈ {8, 16, 32, 64, 128}). Both new heuristics use the same queried groups and architecture-family exclusion as GIDS. GIDS achieves lower observer-macro regret than both heuristics at every tested query budget (0.1266 vs. 0.2263 and 0.1415).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.