acceptodds
Withdrawn from ICLR 2027

GlassSearch-Bench: Task-Adaptive Evaluation and Controlled Rubric Evolution for Smart-Glasses Search Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Smart-glasses search requests require different evaluation criteria: identifying an object, finding a suitable destination, and giving directions impose different obligations. Task-specific rubrics express these obligations, but regenerating criteria for each evaluation can change the checks applied to an unchanged task. We present GlassSearch-Bench, which constructs task-specific rubrics from versioned, reusable atoms to constrain this source of variation. An answer-blind planner extracts requirements; a constructor selects, parameterizes, and narrows applicable atoms, adding local criteria for uncovered requirements. Repeated product comparisons reuse the resulting frozen rubric. Recurring gaps supply candidate library revisions, with validation and human review governing new versions. In an 18-task development study, candidate expansion reduces local criteria from 32 to 18 relative to the eligibility-repaired library and matches all 15 controlled completion targets from five probes. A 90-case replay study documents changes in requirement records and atom selection across repeated construction. A 500-task archive with four system response sets demonstrates the diagnostic value of task-specific criteria: on 432 tasks assessable for all four systems under both reference-answer conditions, every system scores higher on object identification than on either place discovery or wayfinding. GlassSearch-Bench links task-specific evaluation, explicit control of criterion reconstruction, and cumulative library development to actionable product profiles.

Primary area:
datasets and benchmarks
Keywords:
smart glasses, multimodal search, task-adaptive evaluation, atomic rubrics, rubric evolution, benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VstoOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduce the GlassSeaerch-Bench, which evaluates smart-glasses search assistants using task-specific rubrics. Experiments include a 500-task archive covering 4 products, an 18-task development study, and a 90-case construction replay.

Critical strengths

None.

Critical weaknesses
  1. Poor writing. This paper is difficult to follow.

  2. Product comparisons cannot isolate the causes of performance differences. Many factors can affect product's performance, but the experimental design does not explain how the evaluation can be reproducted

  3. There is a lack of the detail about the sources of the human label and validation procedure. And It is necessary to report the agreement between the automated evaluation and the expert evaluation.

Overall, I believe this paper need to be carefully revised.

Questions for authors

See Weaknesses.

Reviewer_m5NVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces GlassSearch-Bench, an evaluation framework and accompanying benchmark for smart-glasses. The core idea is to construct task-specific rubrics by looking at the user query input and context (what the glasses see) but not the agent answer. The authors then present experiments for different tasks and smart glasses.

Critical strengths

S1: The idea to create rubrics "on-the-fly" from a given task is interesting and well-motivated by the diversity of AI applications, such as smart glasses, that are used in a broad range of tasks.

S2: The domain of smart glasses agents is less explored than other AI agent domains and can be offering new interesting, multi-modal benchmarks.

Critical weaknesses

W1: the main criticism I have is that I cannot clearly understand what the contribution, or even the goal of the paper is. The paper starts with the motivation that task-specific rubrics can show variations and the need to make evals more comparable. Then later the story changes to showing benchmarks on smart glasses tasks and what tasks perform better or worse. Is the contribution now a better way to create rubrics or showing what smart glasses can do today?

W2: the writing of the paper is unclear, leaving important parts of the method and contribution underspecified. Large sections of the paper read "AI-generated" (the authors admit they AI but they should edit the paper throughly). In particular the concept of "atom" which is central to the method is never is formally introduced (l. 159 "An atom stores an identity, version, applicability condition, parameter schema, criterion template, and evidence boundary. " - what is an identify? What conditions, schema, templates and boundaries are referred to here? If not a formal definition, please provide illustrative examples for observations, query, context, tools, atoms, rubrics, evidence etc. to make the method clearer to the reader. Section 6 and 7 in particular read like "AI-slop" (l. 424 example "Repeated coverage gaps nominate reviewed library changes." -> what does this mean? the gaps nominate changes but the changes are already reviewed??)

W4: there are no baselines anywhere in the paper, neither is there a baseline of other rubric creation methods (presumably having more variations) nor is there any human study to validate that the presented rubric methods is good / better (compared to what?).

I am not an expert for smart glasses and I might have missed something, but even without understanding the smart glasses domain, the eval methodology should be understandable to the reader.

Questions for authors

Q1: l. 41 :in your motivation be clearer what the source of variations in rubrics are - are these introduced by LLM randomness or by different test cases. If I generate the rubrics for my test data twice, do I get the same rubrics or not?

Q2: in your eval method, is there anything specific to smart glasses? does the method apply to other AI agents?

Q3: Section 3: where do the 500 tasks come from? there is no reference - are thee datasets created by the authors? will they be released?

Q4: the "atom" is a central part of your method but it is never formally defined, and there is not even a good example.

Reviewer_KsXAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GlassSearch-Bench evaluates smart-glasses search responses using task-specific rubrics assembled from reusable atoms. Rubrics are constructed without seeing candidate answers and frozen for subsequent comparisons. The paper reports a 500-task analysis of four products, an 18-task library-development study, and a 90-case reconstruction study.

Critical strengths
  1. The separation of answer-blind construction, frozen evaluation, and later library revision is useful. Explicitly recording template narrowing helps distinguish legitimate task requirements from extra obligations introduced by the evaluator.
  2. The reporting is candid about uncertainty. The paper distinguishes unassessable cases from failures, reports sensitivity analyses, and avoids causal claims from the image-routing comparison. Excluding the pilot with unresolved labels from accuracy claims is appropriate.
Critical weaknesses
  1. The experiments do not establish the added value of atomic construction. In Table 3, reducing local criteria from 32 to 18 shows increased reuse, but not necessarily better evaluation. The 15 controlled targets come from only five development probes. Table 4 measures reconstruction agreement without a matched freeform baseline or downstream score analysis. Since freeform rubrics can also be frozen, the benefit of freezing is not evidence for the benefit of atoms. The larger product study uses a separate v13 protocol and does not close this gap.
  2. The evaluator lacks adequate independent validation. The references are AI-assisted rather than independently adjudicated gold labels, and the response-level pilot cannot support accuracy estimates because its labels remain unresolved. Lower scores after adding references demonstrate sensitivity, not improved correctness. This also limits interpretation of the category gaps, which may reflect differences in evidence availability and scoring criteria alongside product capability.
Questions for authors
  1. Is there evidence on tasks excluded from library development comparing atomic and freeform construction under matched conditions, particularly for requirement coverage, unintended obligations, and downstream score stability?
  2. Can you provide an independently adjudicated response subset that measures evaluator errors and whether reference-induced score changes correct earlier judgments?
  3. Can reviewers inspect complete, sanitized evaluation examples, including the instantiated rubric and aggregation calculation, and clarify which components the v13 study shares with the proposed method?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.