acceptodds
Under review as a conference paper at ICLR 2027

What to Test Next: Interpretable Coverage Gap Discovery in Driving VLMs

Abstract

Driving vision-language models (VLMs) must accurately understand scenes across diverse conditions defined by Operational Design Domains (ODDs), yet verification remains sparse: many slices are missing, making empirical failure rates unreliable. We propose SliceScorer, a deterministic scoring rule for missing-slice recommendation that combines (i) an exposure-based coverage prior to prioritize rare, under-tested regions, and (ii) a neighbor-failure prior that propagates risk from similar tested conditions. SliceScorer is deliberately simple - interpretable, auditable, and conservative - properties essential for safety-critical validation. For stress testing beyond the declared ODD, we embed SliceScorer within SliceNav, an LLM-orchestrated verification pipeline where the model interprets developer queries to select relevant operators (triage, scoring, acquisition, evaluation) and vocabulary extensions, composing verification workflows while keeping all scoring deterministic and auditable. Experiments on three driving VLMs (WiseAD, DriveMM, Cosmos-Reason2-2B) show that SliceNav balances failure-risk capture and recommendation diversity competitively with prior slice-discovery methods, with the coverage–risk tradeoff improving most steeply at larger testing budgets. Ablations confirm both scoring components contribute, and qualitative analysis demonstrates end-to-end workflows from developer query to targeted evaluation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Driving VLM Evaluation, Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2xhkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SliceNav, an LLM-orchestrated verification framework that recommends which untested ODD slices of a driving VLM should be evaluated next. At its core is SliceScorer, a deterministic rule that ranks missing slices by combining a rarity prior, computed from per-dimension marginal frequencies, with a neighbor-failure prior based on similar observed slices. The LLM maps developer queries to verification operators (triage, scoring, ODD extension, acquisition, evaluation), while the ranking itself stays deterministic. Experiments on three driving VLMs report better risk–diversity tradeoffs than adapted slice-discovery baselines, both inside and outside the ODD (Tables 1–2). They also report higher stability than a pure-LLM agent (§4.4).

Critical strengths
  1. Practically relevant instantiation. The paper applies next-test prioritization to driving VLMs over a discrete, ODD-structured slice space (§3.2). This is a practically relevant setting, because VLM validation tables are typically sparse and most ODD combinations remain untested.

  2. Sound design principle with traceable outputs. The design separates LLM-based orchestration from a deterministic scoring function (§3.3, §3.5), so rankings stay reproducible while developers can still interact in natural language. The stability study supports this: pairwise Jaccard is 0.957 vs. 0.027 for a pure-LLM agent (§4.4). Each score also decomposes into an exposure term and a single evidence neighbor (Eq. 7), so every recommendation can be traced to a specific observed slice (Fig. 11). This audit trail is valuable for safety-critical verification.

Critical weaknesses
  1. The in-ODD evaluation (Table 1) does not support the effectiveness of SliceScorer: the large-K results are unclear, and at small K it underperforms the baselines.
  • The K=5000 results need clarification. The in-ODD space has 3,840 slices (§4.1), of which up to 604 are observed (Table 5), yet results are reported at K=5000. Since K exceeds the number of candidates, it is unclear how the methods can still differ (e.g., Regret 0.0258 for SliceScorer vs. 0.0764 for SliceFinder on WiseAD), or how Regret is computed when a method returns fewer than K slices. Without this explanation, the “clean sweep at K=5000” cannot be interpreted as evidence of better ranking.

  • At K ≤ 1000, SliceScorer underperforms the baselines. It has the worst Regret in all six VLM–budget settings and is best in only 1 of 18 metric cells, against all three baselines. The “5 of 9” headline holds only after SliceLine and GP-Mean are labeled “reference” baselines.

  • The ablation does not support the design. The neighbor-only variant matches or beats the full model, e.g., Regret@500 of 0.5583 vs. 0.5618 on WiseAD (§4.1).

  1. The out-of-ODD evaluation (Table 2) rests on an unclear oracle, and SliceScorer is indistinguishable from Random.
  • As described in App. H, the oracle cannot contain extended values. App. H states that scene attributes are assigned by forced choice “from that dimension’s finite in-ODD vocabulary.” Under this procedure, no oracle example can carry an extended value (e.g., tunnel, dusk). It is therefore unclear how failure risk is defined for out-of-ODD slices.

  • The evaluation pool is not characterized. The paper cites a 91M+ space, but the experiment uses an unspecified “shared candidate pool” (App. L). At K=5000, all methods including Random reach nearly identical Regret (e.g., 0.1827 / 0.1825 / 0.1830 on DriveMM), which suggests the pool is small and saturated.

  • There is no meaningful gain over Random. SliceScorer’s Regret differs from Random’s by at most 0.006 in every cell (≤0.001 in 5 of 9), and no variance is reported. Random also has the highest PO in 6 of 9 cells, including all cells at K=5000. The claim of “dominating on risk capture” (§4.2) is therefore not supported.

Questions for authors
  1. What are |S_miss| and the returned list length per method in the in-ODD experiment? How are Regret and ILD computed when a list is shorter than K?

  2. Out-of-ODD oracle. App. H assigns scene attributes from the in-ODD vocabulary only. How do oracle examples obtain extended values (e.g., tunnel, dusk)? How large is the out-of-ODD evaluation pool, and how is it constructed?

  3. Baselines and ablation. What criterion separates “primary” from “reference” baselines, and why does it not also apply to SliceFinder? Given that the neighbor-only variant matches the full model (§4.1), what evidence shows that the rarity prior helps?

  4. (Suggestion for new experiments) Controlled scorer comparison within the agent. The quantitative results (Tables 1–2) evaluate scorers outside the agent and without queries. Section 4.4 compares against a pure-LLM agent that differs in several factors at once and measures only stability. Could you run SliceNav on the 15 queries (App. M), fix the resulting candidate sets, and compare scorers on them, including an LLM that ranks the same candidates with the same evidence, in terms of Regret?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 2xhk (Part 1 of 2)

We thank the reviewer for their time and hard work to help us improve our manuscript. Each critical weakness (W) and question (Q) is answered below in the reviewer's order; tables are labelled A–E and referenced in the text. Please feel free to raise further questions or comments on our responses.


W1.1 "The K=5000 results need clarification."

  • K is a cap. |S_miss| = 3,545 per VLM (3,840 in-ODD slices − 295 observed), so no method can return 5,000 slices.
    • Regret always divides by K, and empty slots count as zero risk.
    • The column becomes K=All (≥|S_miss|). The K=All column is therefore a coverage check, not evidence of better ranking.

W1.2 "At K ≤ 1000, SliceScorer underperforms the baselines."

  • SliceScorer competes on PO, not on Regret or ILD alone. PO = ILD × (1 − Regret) rewards a list that is both risky and diverse, which is the goal of next-test selection.
    • In-ODD at K=500, SliceScorer has higher PO than SliceFinder at DriveMM and Cosmos.
    • Out-of-ODD, the regime the method targets, we lead on PO in 7 of 9 cells among non-Random methods (W2.3).
  • At small K, SliceScorer returns a more diverse list than SliceFinder in all six cells (Table A).
  • The "5 of 9" count included the K=All cells, which are a coverage check (W1.1).
  • In-ODD the seed is dense (the fifth-nearest seed slice is at cosine ≥0.7614 for every candidate); the regime the method targets, values outside the seed's vocabulary, is out-of-ODD (W2).

Table A. ILD at K=500 / K=1000.

MethodWiseADDriveMMCosmos
SliceScorer0.5788/0.59800.5535/0.57410.5852/0.5885
SliceFinder0.5321/0.57240.5220/0.56930.5279/0.5718

W1.3 "The ablation does not support the design."

  • The cited gap is small: WiseAD K=500 Regret 0.5583 (neighbour-only) vs 0.5618 (full), Δ 0.0035. Across the six informative cells, the two differ by at most 0.0070 PO and 0.0100 Regret.
  • Removing the neighbour term lowers PO in every cell (Table B):
  • The neighbour term carries the ranking; rarity is a coverage prior whose cost is at most 0.0070 PO (Table B).

Table B. Ablation, PO at K=500.

VariantWiseADDriveMMCosmos
Full0.25360.36020.3015
Neighbour-only0.25720.36050.3012
Rarity-only0.19430.33880.2521

AuthorsOpenReview ↗

Response to Reviewer 2xhk (Part 2 of 2)


W2.1 "The oracle cannot contain extended values."

  • The forced-choice sentence covers the seed table; App. H's first paragraph states the oracle covers "the base and extended ODD definitions" (App. E.1), and the stored tables carry those values: fog_dense 1,970, dusk 21,545, night_poorly_lit 905, pedestrians 19,965.
  • Out-of-ODD risk is 1 − mean judge score over the slice's oracle examples, as in-ODD.

W2.2 "The evaluation pool is not characterized."

  • The pool is 5⁹ = 1,953,125 candidates, the product of the stored LLM-selected extended domain (pool_size), identical across VLMs. The full extended vocabulary (App. E.1) has 99,655,920 combinations; the paper's "91M+" is a typo and will be corrected. Table 2 is recomputed on this pool for SliceScorer, SliceFinder, SliceLine and Random.
  • The K=5000 compression is oracle coverage, not saturation (Table C): few candidates have oracle support, so an irreducible floor bounds every method's Regret.

Table C. Out-of-ODD pool and Regret floor.

WiseADDriveMMCosmos
Candidates with oracle support1,9201,9201,907
Regret floor, K=5000.58180.85230.8482
Regret floor, K=10000.79090.68970.5995
Regret floor, K=50000.51060.16920.1395

W2.3 "No meaningful gain over Random."

  • SliceScorer competes on PO, not on Regret or ILD alone, and Random is the diversity ceiling, not a competitor (Table 2, †). Random's PO comes from ILD alone: uniform sampling reaches 0.8002 / 0.8000 / 0.8000 at K=500 / 1,000 / 5,000 on every VLM, the expectation 1 − 1/5 for five values per dimension, with no risk prioritisation. Among methods that prioritise risk, we lead on PO in 7 of 9 cells.
  • SliceScorer's Regret interval is disjoint from Random's in 3 of 9 cells, all favouring SliceScorer (WiseAD K=500 and K=1000, DriveMM K=500); the other six overlap (Table D).
  • SliceScorer finds risky slices Random does not (Table D). Note that, hits@500 counts supported slices with risk above the oracle median.

Table D. SliceScorer vs Random, out-of-ODD, K=500 (Random: mean ± std, 10 seeds).

MetricWiseADDriveMMCosmos
SliceScorer Regret [CI]0.9847 [0.9820, 0.9874]0.9969 [0.9959, 0.9976]0.9743 [0.9670, 0.9806]
Random Regret1.0000 ± 0.00021.0000 ± 0.00000.9744 ± 0.0000
SliceScorer hits@5001362
Random hits@5000.100
SliceScorer support@5004.60%5.40%4.20%
Random support@5000.06%0.06%0.06%

Q1. |S_miss|, list lengths, and Regret / ILD for short lists

  • |S_miss| = 3,545. At K=All, list lengths are 3,545 for SliceScorer and Random, 1,000 for SliceFinder, and 3,342 / 3,441 / 3,346 for SliceLine (W1.1).
  • Regret divides by K (empty slots are zero risk); ILD is computed over the returned list.

Q2. Out-of-ODD oracle and pool — see W2.1 (extended values) and W2.2, Table C (pool size, construction, oracle support).


Q3. Primary vs reference baselines; evidence that the rarity prior helps

  • A primary baseline scores every candidate, runs at every budget and out-of-ODD, and gives a per-slice rationale (Table E). SliceFinder meets all three, so the criterion does apply to it.
  • Rarity is what scores never-tested values: out-of-ODD values get an exposure from their nearest observed value, discounted by β (Sec. 3.5). In-ODD its cost is at most 0.0070 PO (W1.3, Table B).

Table E. Baseline capability.

MethodTypeScores every candidateRuns at every budget, in & out-of-ODDPer-slice rationale
SliceScorerRanks untested slices (ours)YesYesYes
SliceFinderFinds high-error slices in test dataYesYesYes
SliceLineFinds high-error slices in test dataNo (drops unmatched slices)YesYes
GP-MeanGeneric acquisition ruleIn-ODD onlyNoNo
RandomUniform samplingYesYesNo

Q4. Controlled scorer comparison within the agent

  • Tables 1 and 2 isolate the scorer: every scorer receives the same pool from the same seed evidence against the same oracle, so only the scoring rule changes.
  • A query only selects which candidates are ranked; Tables 1–2 test the rule on the full S_miss and on a full S⁺(q) (Eq. 9), rather than on the agent's short output lists.
  • Sec. 4.4 measures the system-level property: pairwise Jaccard 0.957 for SliceNav against 0.027 for a pure-LLM agent.

I thank the authors for their rebuttal and the additional results. The response clarifies the evaluation protocol, but my main concern about SliceScorer’s effectiveness for next-test selection remains unresolved.

In-ODD Ranking. The authors argue that SliceScorer should be judged on PO, which combines risk capture and diversity. Even on PO, SliceScorer scores lower than SliceFinder, the authors’ primary baseline, on WiseAD at K=500K=500 and on all three VLMs at K=1,000K=1{,}000. The rebuttal highlights the two PO gains at K=500K=500. In the other four settings, SliceScores underperforms SliceFinder.

The K=AllK=\mathrm{All} Comparison. I agree with the authors that this column serves as a coverage check. The evaluation caps SliceFinder’s output at 1,000 recommendations, whereas SliceScorer returns all 3,545 candidates. Even if the two methods ranked candidates identically, SliceScorer could achieve lower Regret simply by returning more candidates. Section 4.1 still counts these three results toward the reported “5 of 9” wins. These results should be reported separately from the ranking comparison.

Out-of-ODD Ranking. The recomputed results in Table 11 address part of my original concern. For each of the three VLMs, SliceScorer finds more oracle-supported high-risk slices at K=500K=500 than Random finds on average over 10 seeds. However, Random’s reported PO remains higher in six of nine settings in both Tables 2 and 11. These results do not show that SliceScorer outperforms Random in the out-of-ODD setting.

I therefore maintain my rating of 2 (Weak Reject).

AuthorsOpenReview ↗

Response to Reviewer 2xhk (follow-up)

Response to Reviewer 3 (follow-up)

We thank the reviewer for the careful reading and response.

In-ODD ranking

  • The counts are as stated: at K ≤ 1000, SliceFinder leads in four of six settings, by 0.0066–0.0263 PO.
  • Outside the ODD, SliceScorer leads SliceFinder in all 9 settings (Table 11), e.g. Cosmos K=1000 PO 0.1830 vs 0.1500. Across all 15 ranking settings, it leads in 11.
  • In-ODD, every candidate's values have been tested, so SliceFinder has dense evidence. Outside the ODD, SliceScorer transfers failure evidence by similarity; on WiseAD at K=500 and K=1000, SliceFinder captures no oracle risk (Regret 1.0000) while SliceScorer does (0.9847, 0.9892). Next-test selection is the second setting.

The K=All comparison

  • We agree. Section 4.1 and the Table 1 caption will be revised: K=All is reported separately as a coverage check, and the "5 of 9" statement is removed.

Out-of-ODD ranking

  • Random is defined in the paper as the diversity ceiling (Table 2, †), not a competitor; against it, the relevant test is whether a method finds failures, and SliceScorer does in all 9 settings.
  • Random adds nothing to finding the right slices: uniform sampling ignores failure evidence. At K=500 SliceScorer finds 13 / 6 / 2 oracle-supported high-risk slices against Random's 0.1 / 0 / 0, and has lower Regret than Random in all 9 settings (Table 11).
  • Random's PO lead comes from diversity alone: its ILD is 0.80 by construction, against 0.42–0.62 for SliceScorer, and in all six settings where its PO is higher, its Regret is also higher.
  • Among methods that prioritise risk, SliceScorer has the highest PO in 7 of 9 settings.

Hope this clarifies your remaining concerns and helps you alleviate the score!

Reviewer_vWw5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies next-test recommendation for driving VLMs: given a sparse seed validation table over a discrete ODD taxonomy (8 dimensions, 3,840 in-ODD slices), rank the untested slices. An LLM controller dispatches operators (; out-of-vocabulary values get an exposure estimate from the nearest observed value. Experiments on WiseAD, DriveMM and Cosmos-Reason2-2B report Regret@K, ILD@K and their product PO@K against an oracle table built from 15 datasets, with adapted SliceFinder/SliceLine, a GP posterior mean, embedding-kNN and random as baselines. A stability study shows the pipeline is more repeatable than a pure LLM agent.

Critical strengths
  • Prioritising untested slices rather than diagnosing observed ones is a practically relevant framing for ODD-based validation, and the mapping of dimensions is a nice touch.

  • The scorer is cheap, fully specified with all hyperparameters, and the auditability argument is clearly made.

  • Reasonable evaluation infrastructure: a 15-source oracle disjoint from the seed set, with both risk and diversity reported.

Critical weaknesses
  • Seven hand-set hyperparameters, no sensitivity analysis, inconsistent ablation. The scorer depends on α, τ, γ, κ, m, θ, β and a dimension-independence assumption, with no sweep. The ablation is text-only, and its numbers show the neighbor-only variant is better than the full model.

  • Ground-truth reliability and statistical support. Qwen3-30 B produces slice attributes for both seed and oracle, and audits 100 examples; GPT-4.1 judges correctness on text only. Reported gaps between methods are within plausible label-noise effects, and no variance, confidence intervals or seeds are given anywhere.

Questions for authors
  • How is the oracle risk defined for a recommended slice with no oracle examples? What fraction of the top-500 outside-ODD recommendations has oracle support?

  • Can you provide the full ablation table with variance (seeds or oracle bootstrap) and a sensitivity sweep over γ, κ, θ and m?

2 comments
AuthorsOpenReview ↗

Response to Reviewer vWw5 (Part 1 of 2)

We thank the reviewer for their positive assessment and helping us improve the manuscript. Each weakness (W) and question (Q) is answered below in the reviewer's order; tables are labelled A–D and referenced in the text. Please feel free to raise further questions or comments on our responses.


W1. Seven hand-set hyperparameters, no sensitivity analysis, inconsistent ablation

Hyperparameters and sensitivity.

  • All VLMs and budgets share the seven values, and no dataset-specific sweeps were performed (Sec. 4). The sweep below is a robustness check, not a tuning record.
  • We now sweep one hyperparameter at a time, with the rest at default, over 3 VLMs × 3 budgets (Table A).
  • No parameter moves PO by more than 0.0467 (Table A). That maximum occurs only at the lower grid edge κ=0.6; for κ from 0.7 to 1.0, PO changes by at most 0.0002 in every cell.
  • θ is flat: the fifth-nearest seed slice is at cosine ≥0.7614 for every candidate, so θ=0.3 never binds.
  • Dimension independence affects only the exposure term. That term is a product of per-dimension marginals, used as a coverage prior. The failure signal comes from the empirical neighbour map over observed tuples. τ and α act only on the product; they move PO by ≤0.0280 and ≤0.0004 (Table A).

Table A. Largest PO range (max − min over the grid, others at default) per hyperparameter in any VLM–budget cell, in-ODD.

Paramθαγβmτκ
Grid0.2–0.40.5–21–30.6–13–100.4–0.80.6–1
Max PO range0.00000.00040.00580.02470.02640.02800.0467

Ablation.

  • The full ablation is now a table with 95% oracle-bootstrap intervals for three VLMs at K=500 and K=1000 (Table B).
  • Full and neighbour-only differ by at most 0.0070 PO and 0.0100 Regret in any of the six cells.
  • Removing the neighbour term lowers PO in every cell: rarity-only PO is the lowest (Table B), e.g. 0.1943 against 0.2536 at WiseAD K=500.
  • The neighbour term carries the ranking. Rarity is the coverage prior: it ranks under-tested values higher, and it is what gives out-of-ODD values, never tested, an exposure (from their nearest observed value, discounted by β; Sec. 3.5). Its in-ODD cost is at most 0.0070 PO.

Table B. Ablation (in-ODD). R = Regret with 95% CI.

VLMKFull R [CI]Neighbour-only R [CI]Full PONeighbour-only PORarity-only PO
Cosmos5000.4847 [0.4700, 0.4989]0.4855 [0.4709, 0.4997]0.30150.30120.2521
Cosmos10000.3114 [0.3041, 0.3190]0.3045 [0.2973, 0.3123]0.40520.41110.3890
DriveMM5000.3493 [0.3355, 0.3632]0.3490 [0.3350, 0.3629]0.36020.36050.3388
DriveMM10000.2105 [0.2032, 0.2180]0.2088 [0.2013, 0.2162]0.45320.45470.4454
WiseAD5000.5618 [0.5466, 0.5760]0.5583 [0.5428, 0.5732]0.25360.25720.1943
WiseAD10000.3851 [0.3767, 0.3932]0.3751 [0.3667, 0.3835]0.36770.37470.3288

AuthorsOpenReview ↗

Response to Reviewer vWw5 (Part 2 of 2)


W2. Ground-truth reliability and statistical support

Qwen3-30B labels and the 100-example audit.

  • The experts largely agree: raw agreement 0.8263 (4,462 of 5,400 agreeing rater pairs over 900 example × field subjects), with Fleiss' κ 0.2395.
  • κ is low because the labels are mostly correct, not because raters disagree.
    • Verdicts are Y/N/U, and 87.22% of all 3,600 expert verdicts are Y (label correct), so Fleiss' chance agreement is 0.7716. κ only credits agreement above that, leaving 1 − P_e = 0.2284.
    • To confirm that the low κ reflects this skew and not rater disagreement, we also computed Gwet's first-order Agreement Coefficient (AC1) [1], whose chance term does not grow when one answer dominates. AC1 is 0.8039 (Table C), in line with the raw agreement of 0.8263.
  • Even with higher raw agreement, a field with a larger majority share can have lower κ: road_type vs road_condition (Table C).

Table C. Rater agreement, pooled and for two fields.

FieldRaw agreementFleiss' κAC1Majority share
All fields0.82630.23950.803987.22%
road_condition0.75500.31320.701877.75%
road_type0.89670.08840.890594.00%

GPT-4.1 judge on text only.

  • The judge sees the question, the predicted answer and the dataset's reference answer. Almost all judged items are multiple-choice, so the answer is checked against a fixed reference, not an open description.
  • Judged set: 2,950 examples; 2,917 (98.9%) multiple-choice by item-level detection (33 open-ended), 2,909 (98.6%) by Table 5 labels.

Label noise, variance, confidence intervals, seeds.

  • All methods are scored against the same labels and oracle, so no method is scored on cleaner labels than another.
  • SliceScorer is deterministic. It gives identical scores and rankings across instantiations and processes (three configurations), so seeds are not a variance axis for it.
  • The randomness lies in the oracle sample. We now report 95% bootstrap intervals for Regret and PO, and out-of-ODD also for captured risk (1,000 resamples within slice, oracle top-K fixed, seed 12345), as in Table B. Random is mean ± std over 10 seeds.

Q1. Oracle risk for a slice with no oracle examples; top-500 out-of-ODD support

  • A slice with no oracle example has risk 0.0, for every method (set in the metric function).
  • Table D gives the top-500 oracle-support share out-of-ODD.
  • Only 1,907–1,920 of 1,953,125 candidates have oracle support, about 1 in 1,000. Random's 0.06% is a 10-seed mean near that base rate. SliceScorer is 43–55× the pool's base rate, and the only method at ≥4.20% on all three VLMs (Table D).

Table D. Top-500 oracle-support share (out-of-ODD).

MethodWiseADDriveMMCosmos
SliceScorer4.60%5.40%4.20%
SliceFinder0.20%5.60%7.60%
SliceLine9.00%0.00%0.00%
Random0.06%0.06%0.06%

Q2. Full ablation with variance, and a sweep over γ, κ, θ and m

  • Ablation with variance: Table B (W1), with 95% oracle-bootstrap intervals for all three VLMs at K=500 and K=1000.
  • Sweep: Table A (W1); those four hyperparameters give maximum PO ranges of 0.0058, 0.0467, 0.0000 and 0.0264.

References

[1] K. L. Gwet. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1):29–48, 2008.

Reviewer_Tt4nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SliceNav, which is a framework for recommending untested driving scenarios. The core component is SliceScorer, which marks missing slices using marginal-frequency-based rarity and embedding-based neighbour failure scores.

Critical strengths

This is a practical problem. The proposed scoring rule is simple and easy to inspect.

Critical weaknesses

1. Lack of methodological novelty

The main algorithm is an ad-hoc combination of inverse exposure frequency and similarity-weighted neighbour errors. Neither of them is derived from a principled statistical formulation. The agentic component just implements a manually specified LLM skill that routes user queries to predefined tools. No novel learning, planning, or decision-making mechanism is proposed.

2. Unjustified assumption underlying the scoring rule

The neighbor-failure prior assumes that semantic similarity between textual slice descriptions predicts similarity in VLM failure rates, but the authores do not establish or validate this relationship.

3. Potentially misleading experimental comparisons

The baselines SliceFinder and SliceLine are designed for diagnosing observed failure subpopurations, not selecting untested scenarios. Moreover, the evaluation part compares SliceNav against unconstrained LLM generation rather than skill-based agent workflow.

Questions for authors

(1) How Table 1 reports results at K=5000 when the in-ODD space contains only 3,840 slices?

(2) What is the motivation behind the specific formulation of SliceScorer?

(3) Is semantic similarity between slice descriptions actually correlates with similarity in VLM failure rates?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Tt4n (Part 1 of 2)

We thank the reviewer for their hard work and helpful comments. Each weakness (W) and question (Q) is answered below in the reviewer's order; tables are labelled A–C and referenced in the text. Please feel free to raise further questions or comments on our responses.


W1. Lack of methodological novelty

  • Contribution 1: a new problem. We rank untested slices based on current available seed evaluation. Prior methods like SliceFinder and SliceLine take evaluated data with failure labels and return observed subpopulations; neither ranks untested slices or is defined outside the ODD.
  • Contribution 2: a simple decision rule (SliceScorer) for this problem. Each term has one job:
    • Rarity (exposure): Laplace smoothing α stops one observed count from driving a value's exposure to zero. Rarity is a coverage prior that favours under-tested values by construction.
    • Neighbourhood: the cap κ bounds what one extremely similar neighbour can contribute. Removing the neighbour term drops PO at WiseAD K=500 to 0.1943 against 0.2536.
    • Their product ranks a slice high only if it is both rare and close to observed failures.
  • On the agentic component (SliceNav). We do not claim a new learning or planning mechanism within our agentic framework SliceNav. Its contribution is measured: the same LLM with and without the scorer and skill instructions gives pairwise Jaccard 0.957 versus 0.027 across 10 runs × 15 queries (Sec. 4.4). The LLM keeps its strength in flexibly probing the ODD across many value combinations, while the scorer makes the ranking reliable.

W2. Unjustified assumption underlying the scoring rule

  • We tested it directly. For every pair of observed slices (43,365 seed-slice pairs per VLM), we compared how similar their descriptions are with how far apart their failure rates are, using the scorer's own text encoder.
  • In Table A, ρ is the rank correlation between the two (negative means similar slices fail similarly), and the last column is how much the failure-rate gap shrinks for the most similar pairs.
    • The assumption holds on all three VLMs (Table A): the more similar two slices are, the closer their failure rates.
    • It holds most clearly where it matters: the gap keeps shrinking across the high-similarity range, and the scorer only uses such pairs (each candidate's five nearest neighbours have cosine ≥0.7614).

Table A. Description similarity vs failure-rate gap.

VLMρGap shrinks by
Cosmos−0.1836%
DriveMM−0.1139%
WiseAD−0.1948%

AuthorsOpenReview ↗

Response to Reviewer Tt4n (Part 2 of 2)


W3. Potentially misleading experimental comparisons

  • Baselines. No existing method ranks untested slices, so we adapted the two closest (App. F) to score the same candidate pool. Table B shows what each method can do once adapted.

Table B. Baseline capability.

MethodTypeScores every candidateRuns at every budget, in & out-of-ODDPer-slice rationale
SliceScorerRanks untested slices (ours)YesYesYes
SliceFinderFinds high-error slices in test dataYesYesYes
SliceLineFinds high-error slices in test dataNo (drops unmatched slices)YesYes
GP-MeanGeneric acquisition ruleIn-ODD onlyNoNo
RandomUniform samplingYesYesNo

W3. Comparison against a skill-based agent workflow

  • SliceNav is the skill-based workflow. Sec. 4.4 holds the LLM fixed (GPT-5.2, prompts in App. N) and removes the ranking tool and skill instructions; both are compared over 10 runs × 15 queries (App. M) by pairwise Jaccard over full slice tuples.
    • Pure-LLM agent: 0.027. SliceNav: 0.957.
    • Without the tool and instructions the agent loses track of the number and identity of its slices; with them it returns nearly the same set across runs.

Q1. How Table 1 reports results at K=5000 when the in-ODD space has only 3,840 slices

  • K is a cap, not a claim that K slices exist. |S_miss| = 3,545 for every VLM (3,840 in-ODD slices − 295 observed).
    • Regret always divides by K; a shorter list contributes zero risk for its empty slots.
    • The column is renamed K=All (≥|S_miss|).

Q2. Motivation behind the specific formulation of SliceScorer

  • Why a product. It acts as a logical AND: a slice ranks high only if it is both under-tested and near observed failures; an additive form would let either term alone dominate.
  • Auditable. Both terms come from stored counts and embeddings, so each recommendation can be shown with its exposure and its best neighbour.
  • Why closed form. A deterministic scorer makes the agent near-reproducible (Jaccard 0.957, Sec. 4.4).

Q3. Does similarity between slice descriptions correlate with similarity in failure rates?

  • Yes, on observed slices (Table A, W2): ρ = −0.18 / −0.11 / −0.19 (Cosmos / DriveMM / WiseAD) over 43,365 pairs per VLM, and the failure-rate gap shrinks by 36% / 39% / 48% for the most similar pairs.
  • It holds in the range the scorer actually uses (cosine ≥0.7614).

Thank you for your detailed rebuttal and additional analyses. While some of my concerns have been clarified, my main reservations regarding methodological novelty and the justification of SliceScorer remain unresolved. I therefore maintain my original score of 2.

AuthorsOpenReview ↗

Response to Reviewer Tt4n (follow-up)

We thank the reviewer for the follow-up.

Methodological novelty

  • The novelty is the problem and a scorer that solves it. No prior method ranks untested slices; SliceFinder and SliceLine return observed subpopulations and are undefined outside the ODD.
  • The difference shows where it matters: outside the ODD, SliceScorer leads SliceFinder on PO in all 9 settings (Table 11), and on WiseAD at K=500 and K=1000 SliceFinder captures no oracle risk (Regret 1.0000) while SliceScorer does (0.9847, 0.9892).
  • Simplicity is deliberate: a closed-form scorer makes each recommendation auditable and the agent near-reproducible (Jaccard 0.957 vs 0.027 without it, Sec. 4.4).
  • If the reviewer is aware of prior work that ranks untested slices, we would be grateful for the reference so that we can position our contribution against it.

Justification of SliceScorer

  • The core assumption is tested beyond assumed: similar slices fail similarly on all three VLMs (ρ = −0.18 / −0.11 / −0.19 over 43,365 pairs per VLM, Table A of our first response).
  • Each term in SliceScorer has a goal: removing the neighbour term lowers PO in every cell (Table 9); rarity is what scores never-tested values outside the ODD, at a cost of at most 0.0070 PO in-ODD.
  • The product is an AND: a slice ranks high only if it is both under-tested and near observed failures.

Again, if a specific aspect remains open, we would be glad to address it.

Regards,
Authors

Comments on the paper

5 comments
AuthorsOpenReview ↗

Summary of manuscript changes (shown in red in the revised PDF)

  • Table 1, budget column renamed — "K=5000" is now "K=All (≥|S_miss|)". (Tt4n Q1, 2xhk W1.1)
  • Outside-ODD space size corrected — "91M+" is now "≈99.7M" (99,655,920) in Sec. 4 and App. J. (2xhk W2.2)
  • Hyperparameter sweep added — new appendix Table 8: one-at-a-time sweep of all seven hyperparameters; PO changes by at most 0.0467. Pointer added in Sec. 4 after "No dataset-specific sweeps were performed." (vWw5 W1)
  • Full ablation table added — new appendix Table 9: full, neighbour-only and rarity-only for three VLMs at K=500 and K=1000, with 95% oracle-bootstrap intervals. Pointer added in the Sec. 4.1 Ablations paragraph. (vWw5 W1)
  • In-ODD intervals added — new appendix Table 10: Table 1 with 95% intervals for Regret and PO (3 decimals, point values as in Table 1). One sentence added to the Sec. 4 metric definition describing the bootstrap. (vWw5 W2)
  • Outside-ODD analysis recomputed — new appendix Table 11: SliceScorer, SliceFinder, SliceLine and Random on the fixed 5⁹ = 1,953,125-candidate pool, with 95% intervals for Regret, PO and captured risk (hits@K as counts). Pointer added in Sec. 4.2 after the sentence introducing Table 2; Table 2 itself is unchanged. (2xhk W2.2)
Area_Chair_GdovOpenReview ↗

Please respond to rebuttal

Dear Reviewers,

The authors have now responded to the questions and concerns raised by everyone. Could I kindly ask you to review the authors' responses, take a look at the comments from the other reviewers, and actively participate in the discussion?

Thank you all very much for your valuable time and contributions!

Best regards,

The AC

AuthorsOpenReview ↗

Requesting response to rebuttal

Hi Reviewers,

We have provided point-by-point responses to the reviewers' comments and uploaded the revised manuscript, with all changes highlighted in red. We sincerely appreciate the reviewers' valuable feedback and look forward to further comments and discussion.

Regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.