- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SliceNav, an LLM-orchestrated verification framework that recommends which untested ODD slices of a driving VLM should be evaluated next. At its core is SliceScorer, a deterministic rule that ranks missing slices by combining a rarity prior, computed from per-dimension marginal frequencies, with a neighbor-failure prior based on similar observed slices. The LLM maps developer queries to verification operators (triage, scoring, ODD extension, acquisition, evaluation), while the ranking itself stays deterministic. Experiments on three driving VLMs report better risk–diversity tradeoffs than adapted slice-discovery baselines, both inside and outside the ODD (Tables 1–2). They also report higher stability than a pure-LLM agent (§4.4).
Critical strengths
-
Practically relevant instantiation. The paper applies next-test prioritization to driving VLMs over a discrete, ODD-structured slice space (§3.2). This is a practically relevant setting, because VLM validation tables are typically sparse and most ODD combinations remain untested.
-
Sound design principle with traceable outputs. The design separates LLM-based orchestration from a deterministic scoring function (§3.3, §3.5), so rankings stay reproducible while developers can still interact in natural language. The stability study supports this: pairwise Jaccard is 0.957 vs. 0.027 for a pure-LLM agent (§4.4). Each score also decomposes into an exposure term and a single evidence neighbor (Eq. 7), so every recommendation can be traced to a specific observed slice (Fig. 11). This audit trail is valuable for safety-critical verification.
Critical weaknesses
- The in-ODD evaluation (Table 1) does not support the effectiveness of SliceScorer: the large-K results are unclear, and at small K it underperforms the baselines.
-
The K=5000 results need clarification. The in-ODD space has 3,840 slices (§4.1), of which up to 604 are observed (Table 5), yet results are reported at K=5000. Since K exceeds the number of candidates, it is unclear how the methods can still differ (e.g., Regret 0.0258 for SliceScorer vs. 0.0764 for SliceFinder on WiseAD), or how Regret is computed when a method returns fewer than K slices. Without this explanation, the “clean sweep at K=5000” cannot be interpreted as evidence of better ranking.
-
At K ≤ 1000, SliceScorer underperforms the baselines. It has the worst Regret in all six VLM–budget settings and is best in only 1 of 18 metric cells, against all three baselines. The “5 of 9” headline holds only after SliceLine and GP-Mean are labeled “reference” baselines.
-
The ablation does not support the design. The neighbor-only variant matches or beats the full model, e.g., Regret@500 of 0.5583 vs. 0.5618 on WiseAD (§4.1).
- The out-of-ODD evaluation (Table 2) rests on an unclear oracle, and SliceScorer is indistinguishable from Random.
-
As described in App. H, the oracle cannot contain extended values. App. H states that scene attributes are assigned by forced choice “from that dimension’s finite in-ODD vocabulary.” Under this procedure, no oracle example can carry an extended value (e.g., tunnel, dusk). It is therefore unclear how failure risk is defined for out-of-ODD slices.
-
The evaluation pool is not characterized. The paper cites a 91M+ space, but the experiment uses an unspecified “shared candidate pool” (App. L). At K=5000, all methods including Random reach nearly identical Regret (e.g., 0.1827 / 0.1825 / 0.1830 on DriveMM), which suggests the pool is small and saturated.
-
There is no meaningful gain over Random. SliceScorer’s Regret differs from Random’s by at most 0.006 in every cell (≤0.001 in 5 of 9), and no variance is reported. Random also has the highest PO in 6 of 9 cells, including all cells at K=5000. The claim of “dominating on risk capture” (§4.2) is therefore not supported.
Questions for authors
-
What are |S_miss| and the returned list length per method in the in-ODD experiment? How are Regret and ILD computed when a list is shorter than K?
-
Out-of-ODD oracle. App. H assigns scene attributes from the in-ODD vocabulary only. How do oracle examples obtain extended values (e.g., tunnel, dusk)? How large is the out-of-ODD evaluation pool, and how is it constructed?
-
Baselines and ablation. What criterion separates “primary” from “reference” baselines, and why does it not also apply to SliceFinder? Given that the neighbor-only variant matches the full model (§4.1), what evidence shows that the rarity prior helps?
-
(Suggestion for new experiments) Controlled scorer comparison within the agent. The quantitative results (Tables 1–2) evaluate scorers outside the agent and without queries. Section 4.4 compares against a pure-LLM agent that differs in several factors at once and measures only stability. Could you run SliceNav on the 15 queries (App. M), fix the resulting candidate sets, and compare scorers on them, including an LLM that ranks the same candidates with the same evidence, in terms of Regret?
4 comments
Response to Reviewer 2xhk (Part 2 of 2)
W2.1 "The oracle cannot contain extended values."
- The forced-choice sentence covers the seed table; App. H's first paragraph states the oracle
covers "the base and extended ODD definitions" (App. E.1), and the stored tables carry those
values:
fog_dense1,970,dusk21,545,night_poorly_lit905,pedestrians19,965. - Out-of-ODD risk is 1 − mean judge score over the slice's oracle examples, as in-ODD.
W2.2 "The evaluation pool is not characterized."
- The pool is 5⁹ = 1,953,125 candidates, the product of the stored LLM-selected extended
domain (
pool_size), identical across VLMs. The full extended vocabulary (App. E.1) has 99,655,920 combinations; the paper's "91M+" is a typo and will be corrected. Table 2 is recomputed on this pool for SliceScorer, SliceFinder, SliceLine and Random. - The K=5000 compression is oracle coverage, not saturation (Table C): few candidates have oracle support, so an irreducible floor bounds every method's Regret.
Table C. Out-of-ODD pool and Regret floor.
| WiseAD | DriveMM | Cosmos | |
|---|---|---|---|
| Candidates with oracle support | 1,920 | 1,920 | 1,907 |
| Regret floor, K=500 | 0.5818 | 0.8523 | 0.8482 |
| Regret floor, K=1000 | 0.7909 | 0.6897 | 0.5995 |
| Regret floor, K=5000 | 0.5106 | 0.1692 | 0.1395 |
W2.3 "No meaningful gain over Random."
- SliceScorer competes on PO, not on Regret or ILD alone, and Random is the diversity ceiling, not a competitor (Table 2, †). Random's PO comes from ILD alone: uniform sampling reaches 0.8002 / 0.8000 / 0.8000 at K=500 / 1,000 / 5,000 on every VLM, the expectation 1 − 1/5 for five values per dimension, with no risk prioritisation. Among methods that prioritise risk, we lead on PO in 7 of 9 cells.
- SliceScorer's Regret interval is disjoint from Random's in 3 of 9 cells, all favouring SliceScorer (WiseAD K=500 and K=1000, DriveMM K=500); the other six overlap (Table D).
- SliceScorer finds risky slices Random does not (Table D). Note that, hits@500 counts supported slices with risk above the oracle median.
Table D. SliceScorer vs Random, out-of-ODD, K=500 (Random: mean ± std, 10 seeds).
| Metric | WiseAD | DriveMM | Cosmos |
|---|---|---|---|
| SliceScorer Regret [CI] | 0.9847 [0.9820, 0.9874] | 0.9969 [0.9959, 0.9976] | 0.9743 [0.9670, 0.9806] |
| Random Regret | 1.0000 ± 0.0002 | 1.0000 ± 0.0000 | 0.9744 ± 0.0000 |
| SliceScorer hits@500 | 13 | 6 | 2 |
| Random hits@500 | 0.1 | 0 | 0 |
| SliceScorer support@500 | 4.60% | 5.40% | 4.20% |
| Random support@500 | 0.06% | 0.06% | 0.06% |
Q1. |S_miss|, list lengths, and Regret / ILD for short lists
- |S_miss| = 3,545. At K=All, list lengths are 3,545 for SliceScorer and Random, 1,000 for SliceFinder, and 3,342 / 3,441 / 3,346 for SliceLine (W1.1).
- Regret divides by K (empty slots are zero risk); ILD is computed over the returned list.
Q2. Out-of-ODD oracle and pool — see W2.1 (extended values) and W2.2, Table C (pool size, construction, oracle support).
Q3. Primary vs reference baselines; evidence that the rarity prior helps
- A primary baseline scores every candidate, runs at every budget and out-of-ODD, and gives a per-slice rationale (Table E). SliceFinder meets all three, so the criterion does apply to it.
- Rarity is what scores never-tested values: out-of-ODD values get an exposure from their nearest observed value, discounted by β (Sec. 3.5). In-ODD its cost is at most 0.0070 PO (W1.3, Table B).
Table E. Baseline capability.
| Method | Type | Scores every candidate | Runs at every budget, in & out-of-ODD | Per-slice rationale |
|---|---|---|---|---|
| SliceScorer | Ranks untested slices (ours) | Yes | Yes | Yes |
| SliceFinder | Finds high-error slices in test data | Yes | Yes | Yes |
| SliceLine | Finds high-error slices in test data | No (drops unmatched slices) | Yes | Yes |
| GP-Mean | Generic acquisition rule | In-ODD only | No | No |
| Random | Uniform sampling | Yes | Yes | No |
Q4. Controlled scorer comparison within the agent
- Tables 1 and 2 isolate the scorer: every scorer receives the same pool from the same seed evidence against the same oracle, so only the scoring rule changes.
- A query only selects which candidates are ranked; Tables 1–2 test the rule on the full S_miss and on a full S⁺(q) (Eq. 9), rather than on the agent's short output lists.
- Sec. 4.4 measures the system-level property: pairwise Jaccard 0.957 for SliceNav against 0.027 for a pure-LLM agent.
I thank the authors for their rebuttal and the additional results. The response clarifies the evaluation protocol, but my main concern about SliceScorer’s effectiveness for next-test selection remains unresolved.
In-ODD Ranking. The authors argue that SliceScorer should be judged on PO, which combines risk capture and diversity. Even on PO, SliceScorer scores lower than SliceFinder, the authors’ primary baseline, on WiseAD at and on all three VLMs at . The rebuttal highlights the two PO gains at . In the other four settings, SliceScores underperforms SliceFinder.
The Comparison. I agree with the authors that this column serves as a coverage check. The evaluation caps SliceFinder’s output at 1,000 recommendations, whereas SliceScorer returns all 3,545 candidates. Even if the two methods ranked candidates identically, SliceScorer could achieve lower Regret simply by returning more candidates. Section 4.1 still counts these three results toward the reported “5 of 9” wins. These results should be reported separately from the ranking comparison.
Out-of-ODD Ranking. The recomputed results in Table 11 address part of my original concern. For each of the three VLMs, SliceScorer finds more oracle-supported high-risk slices at than Random finds on average over 10 seeds. However, Random’s reported PO remains higher in six of nine settings in both Tables 2 and 11. These results do not show that SliceScorer outperforms Random in the out-of-ODD setting.
I therefore maintain my rating of 2 (Weak Reject).
Response to Reviewer 2xhk (follow-up)
Response to Reviewer 3 (follow-up)
We thank the reviewer for the careful reading and response.
In-ODD ranking
- The counts are as stated: at K ≤ 1000, SliceFinder leads in four of six settings, by 0.0066–0.0263 PO.
- Outside the ODD, SliceScorer leads SliceFinder in all 9 settings (Table 11), e.g. Cosmos K=1000 PO 0.1830 vs 0.1500. Across all 15 ranking settings, it leads in 11.
- In-ODD, every candidate's values have been tested, so SliceFinder has dense evidence. Outside the ODD, SliceScorer transfers failure evidence by similarity; on WiseAD at K=500 and K=1000, SliceFinder captures no oracle risk (Regret 1.0000) while SliceScorer does (0.9847, 0.9892). Next-test selection is the second setting.
The K=All comparison
- We agree. Section 4.1 and the Table 1 caption will be revised: K=All is reported separately as a coverage check, and the "5 of 9" statement is removed.
Out-of-ODD ranking
- Random is defined in the paper as the diversity ceiling (Table 2, †), not a competitor; against it, the relevant test is whether a method finds failures, and SliceScorer does in all 9 settings.
- Random adds nothing to finding the right slices: uniform sampling ignores failure evidence. At K=500 SliceScorer finds 13 / 6 / 2 oracle-supported high-risk slices against Random's 0.1 / 0 / 0, and has lower Regret than Random in all 9 settings (Table 11).
- Random's PO lead comes from diversity alone: its ILD is 0.80 by construction, against 0.42–0.62 for SliceScorer, and in all six settings where its PO is higher, its Regret is also higher.
- Among methods that prioritise risk, SliceScorer has the highest PO in 7 of 9 settings.
Hope this clarifies your remaining concerns and helps you alleviate the score!
Response to Reviewer 2xhk (Part 1 of 2)
We thank the reviewer for their time and hard work to help us improve our manuscript. Each critical weakness (W) and question (Q) is answered below in the reviewer's order; tables are labelled A–E and referenced in the text. Please feel free to raise further questions or comments on our responses.
W1.1 "The K=5000 results need clarification."
W1.2 "At K ≤ 1000, SliceScorer underperforms the baselines."
Table A. ILD at K=500 / K=1000.
W1.3 "The ablation does not support the design."
Table B. Ablation, PO at K=500.