acceptodds
Under review as a conference paper at ICLR 2027

Deep-ARViM: A Perceptual Visibility Metric and Benchmark for Augmented Reality Displays

Abstract

In optical see-through augmented reality (OST-AR), virtual content is optically superimposed onto the real world, so its visibility depends jointly on the rendered foreground and the background light field. This makes visibility prediction a perceptual primitive for AR: without it, systems cannot reliably reason about display brightness, virtual object placement and rendering, or power consumption under real viewing conditions. Yet existing perceptual metrics were designed for image distortion or structural similarity. Here, we present Deep-ARViM (Augmented Reality Visibility Metric), to our knowledge the first learned perceptual metric explicitly designed to capture content visibility in OST-AR. We collect a large psychophysical dataset with two complementary tasks: 67,984 two-alternative forced-choice (2AFC) judgments and 21,027 visibility matching responses from more than 550 observers, spanning diverse egocentric backgrounds, foreground contents, and luminance conditions. We formulate visibility as a distance between the composited image and the standalone background, and evaluate both hand-crafted features and adapted pretrained vision backbones on their ability to predict human responses. Our best learned models achieve on par performance with an individual human observer in predicting others' 2AFC responses, recover a graded visibility scale that aligns with independent subjective ratings, and transfer zero-shot to quantitative visibility matching. Scientifically, these results suggest that AR visibility is well described by a shared latent perceptual scale that can be recovered from inexpensive binary supervision and generalized across psychophysical tasks. Practically, Deep-ARViM provides a standardized benchmark and a path toward perceptually grounded AR display adaptation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Augmented Reality, perceptual metrics, visibility prediction, image quality, display adaptation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_PXxLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Deep-ARViM, a perceptual visibility benchmark and learned metric for optical see-through AR. The dataset contains 67,984 pairwise visibility judgments and 21,027 visibility-matching responses. Visibility is modeled as a learned feature distance between a background image and its AR composite, using pretrained vision backbones adapted through channel weighting, LoRA, or full fine-tuning. Models trained on pairwise choices are evaluated on those choices, subjective visibility ratings, and brightness-matching responses without matching-task supervision. A separate 17-participant optical study tests transfer from simulated compositing to real additive light.

Critical strengths
  1. The paper addresses a useful perceptual problem with a substantial human-judgment dataset. Combining pairwise choices with visibility matching provides complementary evidence about relative visibility and the brightness needed to achieve comparable visibility. This gives the benchmark value beyond a single prediction task.

  2. The evaluation connects binary supervision to other perceptual judgments. Models trained only on pairwise choices improve brightness-matching predictions, and the web-trained DINOv2-LoRA model reaches 79.1% consistency-weighted accuracy in the optical study, compared with 68.6% for the learned low-level baseline (Appendix D). These results support the usefulness of the learned score across the tested protocols.

Critical weaknesses
  1. Generalization to unseen content is not sufficiently established. Appendix C.2 describes an 80/20 split over stimulus pairs and selects checkpoints using validation performance, but does not clarify whether foreground images or background scenes are shared across splits. Pair-level separation alone does not establish content-level separation, and no separate final test set is described. This matters for a metric intended to handle arbitrary foreground–background combinations.

  2. The modeling contribution mainly adapts existing perceptual-distance learning and pretrained backbones, so much of the paper’s value rests on the benchmark and transfer evidence. That evidence remains limited to mostly static stimuli with centered, fixed-size foregrounds, plus a small optical study. It supports a metric for the tested conditions, while broader claims about general AR visibility and practical display adaptation remain less established.

Questions for authors
  1. Will the dataset (images, metadata, human annotations) and code be released, including licensing details for Ego4D/Aria backgrounds and Gemini-generated foregrounds?

  2. Can the authors clarify overlap across the training and validation splits at the levels of foreground images, background scenes, and source videos? Are results available on a content-disjoint test set that was not used for hyperparameter or checkpoint selection?

  3. Were the stimuli used for visibility matching and the optical study excluded from 2AFC training? Please distinguish transfer to new judgments from transfer to unseen content, and provide uncertainty estimates for the transfer results by resampling stimuli or participants.

Reviewer_uqzUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents a novel image metric named Deep-ARViM, which is claimed to be the first learning metric for capturing content visibility in OST-AR. To support this task, the authors collect a large psychophysical dataset in sRGB domain with labels provided by humans. The authors found that the learned models can generalize from this specific task to other psychophysical tasks.

Critical strengths
  1. The paper is claimed to be the first to propose a specific metric to capturing content visibility in OST-AR and it recognizes that existing image metrics (e.g., PSNR, SSIM and LPIPS) are not suitable for this task.

  2. The authors collected a large dataset with manual annotation to verify the effectiveness of the proposed metric. Additionally, the collecting procedure sounds valid and scientific.

  3. The authors systematically explored several vision backbones under three adaptation regimes, and the performance of the best model matches the the individual human observer baseline. Additionally, it also demonstrated strong zero-shot transfer capabilities to continuous quantitative visibility matching.

Critical weaknesses
  1. The dataset is built in sRGB domain, and experiments are conducted in sRGB domain, instead of HDR domain. In real-world AR, ambient illumination spans an extreme dynamic range since humans can perceive a wide range of luminance, far more than sRGB domain.

  2. All images in the dataset, as pointed by the authors, are synthesized while not captured by real AR devices, where a domain gap might exist. It is better if the authors can test their proposed metrics on HDR images captured by real-world AR devices to verify the effectieness of the proposed metric (and compare with HDR metrics, e.g., HDR-VDP).

  3. Among synthesized images in the dataset, each image was rescaled to occupy 25% of the background area and placed at the center of the background. What could happen if the foreground images were rescaled to occupay 50% or more of the background area, or placed in other regions of the background?

  4. There is no closed-loop user evaluation demonstrating that integrating Deep-ARViM into an actual AR rendering/backlight control pipeline delivers tangible gains.

Questions for authors
  1. Since the metric does not take the clean foreground as an input, how can you guarantee the model distinguishes between "content that is clearly legible" and "content that is unreadable yet creates a large visual disturbance due to background interference"?

  2. Could you please test the metric on HDR data and see whether it also works, especially on images with extreme contrast (e.g., peak luminance > 1000 nits)?

Reviewer_ML8jOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

A new dataset Deep-ARVim and learned perceptual metrics are introduced for predicting content visibility in optical see-through augmented reality (OST-AR). Existing image quality datasets and metrics are not suited for AR visibility, which depends on foreground-background interactions in additive compositing. The authors collected a large psychophysical dataset comprising from over 550 observers using simulated AR stimuli constructed from egocentric video backgrounds and diverse foreground content. Visibility is formulated as a distance between the composite image and the standalone background in the latent space of a pretrained vision backbone. They adapt seven backbones under three training regimes, and the best model achieves high accuracy matching individual human observers.

Critical strengths
  1. The most important contribution is the construction of a large-scale subject-rated dataset with well-thought stimulus design (diverse backgrounds, foreground types, luminance levels) and quality-control procedures (catch trials, consistency filtering). I believe the dataset itself, once made public, will make strong and lasting impact in the field.
  2. The best model matches individual human performance on 2AFC and transfers zero-shot to visibility matching. The analysis of failure modes provides meaningful insights.
  3. The computational cost of the proposed models seems to remain at a low level, making it feasible for real-time display adaptation. The content-placement example demos a concrete application.
Critical weaknesses
  1. The limitations of simulated compositing, and the effect of the gap from the real-world is unclear. The dataset uses additive blending in linear RGB space. How good is such an approximation from real optics? The luminance range is also far narrower than the adaptation range of the visual system. The experiment uses static, coplanar images. In real AR, relative depth and motion cues affect visibility.
  2. The model design may be reasonable as a first attempt on the dataset, but the novelty is moderate.
  3. The metric provides a visibility score but does not address how to balance visibility against other objectives.
  4. Only generic IQA metrics are compared. How about AR-specific metrics, including those mentioned in “Related Work”?
Questions for authors
  1. Add AR-specific metrics in the comparison if possible.
  2. If low-level cues add little and deep features alone also fall short, then what types of cues are the most relevant? Some empirical investigations or insights would be helpful.
  3. The haploscope uses static stimuli. How do you expect the metric to perform under dynamic lighting conditions or with real optical see-through displays?
  4. The model is trained on simulated images. How sensitive is it to the accuracy of the additive blending model? How about incorporating a more physically accurate rendering?
  5. The current stimuli use uniform luminance multipliers. Would the model generalize to varying transparency?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.