acceptodds
Under review as a conference paper at ICLR 2027

InspectionBench : Evidence-Grounded Evaluation of Agentic Industrial Inspection

10 views

Abstract

With the rapid development of industrial intelligence and robot-assisted inspection, reliable physical-to-operational coupling from evidence acquisition and physical verification to downstream operational decisions has become a key bottleneck for deploying autonomous inspection and predictive-maintenance agents in real-world industrial environments. Existing benchmarks remain limited to static or single-modality observations, hand-specified scenarios, or outcome-oriented evaluation that fails to determine whether the required evidence was actually acquired. We introduce **InspectionBench**, an execution-grounded benchmark for agentic physical AI in industrial inspection, comprising 4,120 canonical episodes generated from 1,552 grounded worlds and 21 scenario templates across six industrial asset types, in which agents acquire visual, operational, and telemetry evidence with explicit provenance and execute inspections before making downstream operational decisions. A world-to-scenario compiler automates matched-episode generation through parameterized control of evidence visibility, modality and tool access, temporal validity, and guidance while preserving the underlying world and gold decision, adjudicated by domain experts where needed. We evaluate five complementary capabilities across five frontier models, finding that correct-action rates exceed evidence-supported success in 20 percentage points and exact physical-constraint identification up to 37.2 points, highlighting capability gaps that final task success alone can obscure. Code and evaluation artifacts are available in https://anonymous.4open.science/r/InspectionBench-3B71/README.mdCode.

Primary area:
datasets and benchmarks
Keywords:
Industrial Inspection, Embodied AI, Physical AI, Agentic AI, Robotics, Tool-Using Agents, Evidence Grounding, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Mgp9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper introduces a benchmark for industrial inspection agents that integrates multimodal evidence acquisition, robot tool use, and operational decisions. It defines and evaluates five agent abilities and reveals substantial gaps between terminal decision accuracy and evidence-grounded success.

Critical strengths
  1. The proposed environment and benchmark are a meaningful integration of agent-based physical evidence acquisition and downstream operational decision-making in a real-world application field.
  2. The scenario generation is well-controlled, and the design of the trace-level evaluation is rational for diagnosing agent failures beyond terminal accuracy.
Critical weaknesses
  1. The paper acknowledges that evidence delivery does not establish correct reasoning, which limits how strongly the reported TDA–GSR gaps can be interpreted as evidence-grounding failures. Therefore, the bench seems to be only for an inspection application evaluation rather than a solid agent capability test.
  2. The agent evaluation protocol and empirical controls are insufficiently specified to fully support reproducible and discriminative model comparisons.
Questions for authors
  1. What is the detailed agent architecture or the harness used in the evaluation? Does the benchmark environment support different or even customized agent harnesses?
  2. How often do successful episodes contain explicit references to the relevant observations? For temporal grounding, how does the metric distinguish actual reliance on outdated observations from legitimate decisions not to re-observe under resource constraints?
  3. Could the authors distinguish constructed, attempted, successfully executed, excluded, and scored episodes for each model and capability?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Mgp9 -1/3

Thank you for your constructive review. Please find our point-by-point response across three comments below.

W1 (Part 1) & Q2 (Part 1). Interpreting GSR, Decomposing the TDA–GSR Gap, and Explicit Observation References

[W1] "The paper acknowledges that evidence delivery does not establish correct reasoning, which limits how strongly the reported TDA-GSR gaps can be interpreted as evidence-grounding failures. Therefore, the bench seems to be only for an inspection application evaluation rather than a solid agent capability test." [Q2 Part 1] "How often do successful episodes contain explicit references to the relevant observations?"

We agree that successful evidence delivery does not by itself demonstrate that an agent used the delivered evidence correctly. We clarified the interpretation of GSR and Δ_TDA-GSR in Sec. 4 (p. 8) and Sec. 5.1 (pp. 9–10): GSR is a trace-verifiable minimum condition combining terminal correctness with successful delivery and provenance of the required physical evidence, rather than a direct measure of internal reasoning. While GSR = 1 is a necessary floor, GSR = 0 when TDA = 1 certifies that a correct action was taken without receiving the required physical observation making the 23.2–29.8 pp TDA–GSR gaps across four of five models (Fig. 6, p. 10) a conservative lower bound on ungrounded commitments.

Alongside GSR, we report four complementary evaluations (Sec. 3.2, p. 6; Sec. 5.1, pp. 9–10; App. B.1, I.7–I.8, pp. 21–22, 34, 38–40):

  • Explicit observation references & validated attribution (Q2 Part 1; EORR & VEA, App. B.1, Table 7, p. 21; App. I.7, p. 34): Because GSR's coherence check accepts any delivered observation of the required modality when no ID is named, we audited stored traces directly via Explicit Observation Reference Rate (EORR), the share of GSR-successful episodes whose output explicitly cites a delivered, asset-matched physical observation ID:
    • Citation-enabled protocol (2,500 episodes; 500/model): 100.0% for Claude Opus 5.5 (260/260), GPT-6-Astra (327/327), Gemini 3.1 Pro (365/365), and Qwen3.5-397B (378/378), and 97.5% for DeepSeek V4 Pro (318/326; 95% CI [95.8%, 99.1%]). Across all 1,656 GSR-successful episodes, 0 episodes cited an undelivered or wrong-asset observation ID.
    • Original protocol without a citation field (11,940 episodes): Explicit IDs appeared in free-text rationales in only 1 of 6,093 GSR-successful episodes, showing that agents reliably name observation IDs when prompted by the schema and almost never otherwise.
    • Validated Evidence Attribution (VEA): Requiring the decision to be consistent with cited structured observations retains 255–369 of 500 episodes per model (−2 to −12 below GSR), while exposing that 1.6%–11.8% of decisions still contradict the agent's own cited evidence.
  • Information-conditioned action admissibility (A_I, IJA & Strict GSR, App. I.7, Tables 23–24, pp. 38–39): When physical readings are withheld (DIGITAL_ONLY, 255 COMMIT-gold worlds/model), 47.1% of DeepSeek V4 Pro and 32.5% of Qwen3.5 decisions are unsupported commitments, whereas GPT-6-Astra (100%) and Claude Opus 5.5 (97.6%) choose justified ESCALATE. Pairing this with each model's matched-world ΔCOMMIT rate (FULL minus DIGITAL_ONLY: +0.36 to +0.73 for four models vs. +0.02 [−0.02, +0.05] for DeepSeek) separates evidence-dependent caution from default escalation.
  • Behavioural sensitivity to delivered readings (App. I.7, Table 25, p. 40): Across 100 matched world pairs per model and regime where only the delivered physical reading moves out of band while IoT, work order, and technician context stay fixed (921/1,000 complete pairs; 79 unparsed pairs excluded rather than counted as no-flips), P(ESCALATE) rises by Δ = +0.43 to +0.90 (exact McNemar p ≤ 3 × 10⁻⁸ in every cell; only 1 reverse flip in 921 pairs).
  • Semantic claim support (App. I.8, p. 34; Table 26, p. 40): Across 30,912 extracted factual claims, deterministic, DeBERTa-v3-large NLI, and hybrid verifiers alongside cross-family LLM judges (GPT-6-Astra and Kimi K3.1 Pro) show 56%–57% [0.42, 0.71] of claims fully supported by delivered observations on the HT-weighted random stratum (n = 75; 35%–37% beyond the evidence; inter-judge κ = 0.84).
AuthorsOpenReview ↗

Response to Reviewer Mgp9 -2/3

W1 (Part 2). Capability-Decomposed Evaluation & Empirical Longitudinal Grounding

[W1 (Continued)] "...Therefore, the bench seems to be only for an inspection application evaluation rather than a solid agent capability test."

More broadly, InspectionBench is designed as a capability-decomposed benchmark for the physical-to-operational agent lifecycle rather than a narrow application test or generic tool-calling benchmark. It complements τ/τ²-bench (tool-agent-user policy adherence) and AssetOpsBench (enterprise asset workflows) by isolating the core operational bottleneck motivated in Sec. 1 (p. 1): industrial telemetry generates far more anomaly alerts than operators can dispatch crews for (with field deployments such as ANYbotics reporting >70% false-positive alarms when unverified physically). An IoT-only agent (DIGITAL_ONLY) that acts without physical grounding either floods operations with false escalations or makes unverified commitments (as seen in our unsupported-commitment rates and 23.2–29.8 pp TDA–GSR gap).

Evaluating agents across our 24,601 replayed operational records and longitudinal compressor/plant failure trajectories (MetroPT-3 / FutureSim chronological replay; App. N, N.1, pp. 39–40, Figs. 14–15) empirically demonstrates why each of our five capabilities is necessary to govern downstream forecasting and intervention reliability:

  1. (A) Evidence Grounding & False-Alarm vs. Fault-Visibility Trade-off (App. N, Fig. 14, pp. 39, 45): Across 76 false incidents and 11 documented true failure incidents in the chronological replay, comparing alert-time telemetry (R1e), decision-time telemetry (R1m), and robot-acquired physical inspection framing (R2r) against oracle-survey (R2o) and rule baselines (R3) makes the operational trade-off explicit: physical verification suppresses unnecessary escalations on false telemetry alarms while preserving high fault visibility on true incidents (Fig. 14, p. 45).
  2. (B) Active Acquisition (ADA 0.985–1.000, ASA 0.911–0.933, TDA_post 0.167–0.727; Sec. 5.1, p. 10; App. I.4, Table 21, p. 36): Recognizes when initial telemetry or acoustic evidence is ambiguous and selectively triggers complementary acquisition before a developing fault crosses failure thresholds, while exposing that post-acquisition interpretation (TDA_post, where models over-escalate in 36%–82% of adequate-evidence episodes) is the primary bottleneck.
  3. (C) Procedural Execution (ordering up to 100% vs. coverage down to 0.12; Sec. 5.2, p. 10; App. I.5, p. 33): Ensures multi-step diagnostic and safety protocols are substantively completed rather than superficially ordered before clearing an asset back to service.
  4. (D) Relational Physical Grounding (37.2 pp gap between terminal accuracy and exact physical-constraint recovery; Sec. 5.2, Fig. 7, p. 10; App. M, Fig. 13, p. 38): Certifies via a MuJoCo-backed admissibility verifier that the agent identifies the governing physical constraint (reach, clearance, joint/collision, stability, energy) or active enterprise work-order lockout (App. I.2, Table 19, p. 34) rather than guessing the right action for the wrong physical reason.
  5. (E) Temporal Grounding & Chronological Failure-Forecast Evolution (App. B.5, Table 8, p. 24; App. N.1, Fig. 15, pp. 40, 45): Directly governs longitudinal monitoring across chronological streams. In our February 2020 MetroPT-3 chronological replay across five 24-hour windows (App. N.1, Fig. 15, p. 45), comparing IoT-only (Condition A) against budgeted physical gauge requests (Condition C, 115 valid hourly 6-hour failure forecasts per arm) shows that once the daily observation budget is spent, forecast probabilities drift upward across all windows (Spearman ρ > 0 in all 10 window-condition pairs, reaching ≥ 0.9 in 29/115 forecasts under Condition C vs. 0/115 under Condition A despite zero true failures). This directly motivates Capability E's resource-aware temporal validity evaluation (TVR^RA = 97.4%–100% for four models vs. 49.6% for Qwen, which commits on stale readings in 62.4% of budget-exhausted visits; Table 8, p. 24).
AuthorsOpenReview ↗

Response to Reviewer Mgp9 -3/3

W2, Q1 and Q3. Reproducibility, Reference Harness, Empirical Controls and Episode Accounting

W2: "The agent evaluation protocol and empirical controls are insufficiently specified..." Q1: "What is the detailed agent architecture or the harness used in the evaluation? Does the benchmark environment support different or even customized agent harnesses?" Q3: "Could the authors distinguish constructed, attempted, successfully executed, excluded, and scored episodes for each model and capability?"

We expanded Sec. 4–5 (pp. 7–9), App. A–B (pp. 17–24), App. K–L (pp. 35–37; Tables 30–34, pp. 42–43), and the Reproducibility Statement (p. 11) to specify all protocol details, controls, and step-by-step reproduction commands (REPRODUCE.md).

Q1. Harness and agent architecture (App. A, Fig. 8)

  • Reference harness: Fig. 8 (p. 17) illustrates our deterministic two-call loop. Turn 1 provides asset context and regime-masked tool names (25 FULL, 23 PHYSICAL_ONLY, 22 DIGITAL_ONLY), executing the first 8 requested calls auto-bound to asset_id. Turn 2 returns structured JSON observation records (status, observation_id, observation_hash, modality, payload, observed_at) for the terminal verdict and citations. Verbatim prompts, a full episode transcript, and frozen settings are in Listings 1–3 (pp. 18–19) and Table 30 (p. 42).
  • Custom harnesses: We expose 67 tools across 7 FastMCP servers with full JSON schemas (App. A.5, p. 19) registered in AssetOpsBench's registry so any MCP-compatible SDK or multi-turn runner can attach over stdio. Because exposing full schemas in a 1-round, 8-call loop shifts agent behaviour (App. A.7, Table 6, p. 20), custom-harness runs should be compared with each other rather than Fig. 5.
  • Call-lifecycle audit (Table 5, p. 20): Auditing all 14,440 traced episodes confirms 0 calls failed on arguments/syntax and 0 of 26,612 executed gold-determining calls failed.

W2. Capability scoring and empirical controls (App. B, K.5–K.7, L.1)

  • Deterministic scoring: Every metric across Capabilities A–E is a pure Python function of the response and hash-chained trace with no LLM judge (missing/invalid outputs score 0; formal definitions in App. B.1–B.5, pp. 21–24).
  • Pre-run empirical controls & reproduction: In App. B.6, K.5, and L.1 (pp. 24, 35–37), we document our fixed reference policies, matched regimes holding world/gold fixed, paired world-cluster bootstrap CIs (B = 10,000, Holm-corrected), and pre-registered 237-episode repeat audit (Table 34, p. 43), ranking models only where paired CIs exclude zero. To easily verify all results, we invite the reviewer to follow our 5-step walkthrough in App. K.6 (p. 36) and REPRODUCE.md in the anonymous repo, which provides direct scripts (compare_runs.py) for API-free CPU rescoring, episode regeneration, and full re-runs (compute/cost in Table 32, p. 43).

Q3. Episode accounting (App. K.2, Table 31, p. 42)

  • Episode breakdown: Table 31 (p. 42) tracks every episode per capability and model: of 4,120 constructed canonical episodes, a pre-run script excluded 49 (observationally identical seeds in A/E or unreachable route gold in D-enterprise), leaving 4,071 eligible episodes attempted and scored for every model with 0 remaining infrastructure failures (invalid/missing outputs remain in the denominator and score 0).

Q2 (Part 2). Temporal Grounding under Resource Constraints

Q2 (Part 2): "For temporal grounding, how does the metric distinguish actual reliance on outdated observations from legitimate decisions not to re-observe under resource constraints?"

Thank you, this question exposed a real gap, which we audited directly from our frozen traces without re-running models (Sec. 5.3, p. 10; App. B.5, Table 8, pp. 23–24; App. F.5, Table 15, p. 29). Because physical reads draw on a sequence budget β = n − 1 and are refused once spent, original TVR did not separate stale reliance from forced inability to re-observe (we also corrected App. F.5 on SEQ-06001 ep. 2). Classifying all 4,690 repeated episodes (469 sequences/model; Table 8, p. 24) separates fresh reads (1,949), budget-feasible stale reliance (0 of 542), budget-exhausted safe abstentions (ESCALATE/ABORT, 2,193), and budget-exhausted commitments (483, concentrated in Qwen at 62.4% [472/757] and DeepSeek at 2.4% [11/463], with 465/483 rationales explicitly citing earlier visits). We therefore added Resource-Aware TVR (TVR^RA), which exempts budget-exhausted abstentions while penalizing ungrounded COMMITs, shifting repeated-episode TVR from 41.7–50.6% (19.2% Qwen) to 97.4–100.0% for four models versus 49.6% for Qwen (Table 8, p. 24).

If these answers resolve your concerns, we would be grateful if you could reflect them in your assessment.

Reviewer_o43zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces InspectionBench, a benchmark for evaluating industrial inspection agents across evidence acquisition, physical verification, and downstream operational decisions. It combines facility images, rendered observations, and replayed operational records with controlled scenario generation. It also evaluates five multimodal models to reveal gaps between correct final actions and evidence-supported execution.

Critical strengths

-The benchmark connects inspection evidence to downstream operational actions and distinguishes final-action correctness from successful evidence acquisition. This provides a useful perspective on agent reliability beyond conventional task-success evaluation.

-The scenario compiler varies evidence availability and tool access while preserving the underlying world and correct decision. This design supports interpretable comparisons of agent behavior under different observation conditions.

-The evaluation separately examines evidence grounding, acquisition, procedural execution, physical constraints, and temporal validity. These diagnostics expose failure patterns that a single aggregate success score would obscure, while deterministic scoring makes the results easier to audit.

Critical weaknesses

-The annotation pipeline relies on relatively older MLLMs, Gemini 2.5 Flash and GPT-4o-mini, for initial labeling and cross-checking. The authors should demonstrate that the resulting annotations are sufficiently reliable to support the benchmark’s evaluation. This concern is reinforced by the low reported agreement on asset identity and gauge values. Although expert adjudication is described, additional validation with stronger annotation models and complete reporting of the human-review results would strengthen confidence in the ground truth.

-Correct behavior under incomplete evidence is not sufficiently defined. The benchmark preserves the same gold action across evidence-access regimes, even when the information required to justify that action becomes unavailable. For example, DIGITAL_ONLY necessarily produces zero grounded success when physical evidence is required. The evaluation should more clearly distinguish the action determined by the complete world state from the action justified by the information available to the agent.

-The input interface may confound tool-use difficulties with inspection capability. Section 4 states that agents receive tool names without schemas or argument specifications, but does not clearly explain how valid tool invocations are discovered. Since failed calls cannot provide evidence and therefore reduce grounded success, some failures may reflect interface ambiguity rather than limitations in evidence acquisition or physical reasoning. A comparison with complete tool specifications would help clarify what the benchmark actually measures.

Questions for authors

-When a matched evidence regime removes the physical observation required to justify a COMMIT decision, should an agent that appropriately chooses ESCALATE or ABORT receive credit? How does the evaluation distinguish justified caution from an incorrect decision or an unsupported commitment?

-Could the authors provide a complete agent-visible interaction specifying the initial input, tool signatures, returned observations, and required output format? In particular, does read_gauge return only an image, or does it also expose a numerical reading or other interpreted information?

6 comments
AuthorsOpenReview ↗

Response to Reviewer o43z -1/3

Thank you for your constructive review. Please find our point-by-point response across three comments below.

W1. Annotation Reliability and Human Review

[W1] Reliance on older MLLMs (Gemini 2.5 Flash and GPT-4o-mini) for initial labeling and cross-checking; low reported agreement on asset identity and gauge values; need for stronger annotation models and complete reporting of human-review results.

We agree that relying on older models for initial labeling and cross-checking could affect data generation reliability. Following the protocol in IndustryBench §4.2 (arXiv:2605.10267), we have revised Sec. 3.1 (p. 5) and App. C–D (pp. 23–24) with three updates:

  1. Re-annotation with current-generation frontier VLMs: We re-annotated all 116 real-image sequences across three current-generation VLMs from distinct vendors (Claude Opus 5.5, GPT-6-Astra, and Gemini 4 Argon) under a single frozen prompt and schema (348 requests; 88/116/110 schema-valid; App. C). Using field-appropriate metrics, three-model Krippendorff's α\alpha is 0.91 for coarse instrument type, 0.91 for normalized units, 0.56 for readability, 0.49 for recommended action, 0.42 for category, and 0.48 (interval α\alpha, 95% CI) for continuous gauge values (Table 8, App. D, p. 24).
  2. Blinded human practitioner audit: Four instrumentation and control practitioners (10 years combined experience, 120 person-hours) built and adjudicated the benchmark. On an independent, blinded 30-scenario audit across four ordinal dimensions (operational realism, difficulty calibration, evidence sufficiency, and gold-action correctness), three reviewers achieved strong agreement (Krippendorff's α=0.84\alpha = 0.84, 95% CI; 24 accepted, 5 revised, 1 rejected; App. E, p. 24), confirming high human consensus on the decision contracts.
  3. Zero impact on compiled-world ground truth (0.0 pp0.0\text{ pp} variance): To verify whether residual VLM annotation noise alters evaluation conclusions, we audited how annotations feed into episode scoring. While the 116 facility photos ground visual appearance and seed PerceptionGauge scenes, all compiled-world episodes (1,280 worlds; 100% of Family A, D-physical, and E) compute ground-truth gauge values and gold actions a∗a^{*} deterministically via sample_world(seed, cell) (Sec. 3.2, p. 5), never from VLM labels on the 116 photos. Consequently, VLM annotation variance shifts compiled-world TDA and GSR by 0.0 pp0.0\text{ pp} and leaves all model rankings 100% unchanged.
AuthorsOpenReview ↗

Response to Reviewer o43z -2/3

W2 & Q1. Incomplete Evidence: Distinguishing Justified Caution from Unsupported Commitments

[W2 & Q1] Preserving complete-world gold actions under incomplete regimes (DIGITAL_ONLY); distinguishing justified caution (ESCALATE/ABORT) from incorrect decisions or unsupported commitments.

We agree that the action determined by the complete world state and the action justified by the observations available to an agent are distinct, and that an inability to observe a withheld modality should not be conflated with an incorrect inspection decision.

W2 & Q1 (Part 2): Distinguishing Justified Caution from Incorrect Decisions or Unsupported Commitments

Q1 (Part 2): "How does the evaluation distinguish justified caution from an incorrect decision or an unsupported commitment?"

1. Original Design: Why Complete-World Gold Is Preserved

In our original design, the gold action a* is defined by the ground-truth physical state (e.g., actual gauge reading, true leak state). Across matched regimes (e.g., DIGITAL_ONLY vs. FULL), the compiler holds the physical world and a* fixed while restricting observations (Sec. 3.2, p. 6). This isolate-and-stress design tests whether missing observations cause an agent to inappropriately COMMIT without required physical evidence yielding GSR = 0 by design under DIGITAL_ONLY when physical verification is required.

2. Revision: Information-Admissible Actions, Strict GSR, and Behavioural Sensitivity

To explicitly evaluate justified caution alongside over-confidence, we revised Sec. 3.2 (p. 6), Sec. 5.1 (pp. 9–10), and App. B.1, I.7 (pp. 21–22, 34, 38–40) to introduce:

  • Information-Admissible Action Set (A_I): Defines the actions justified by delivered observations (Sec. 3.2, p. 6; App. B.1, p. 21). When required physical safety verification is absent from the delivered observations, COMMIT is excluded from A_I. Choosing an admissible action is tracked via Information-Justified Action Rate (IJA; App. B.1, p. 21; App. I.7, Table 23, p. 38).
  • Strict Grounded Success Rate (Strict GSR): Requires both (i) acquiring all required evidence items for the underlying world state and (ii) selecting an action within A_I (Strict GSR = GSR ∧ â ∈ A_I; App. B.1, p. 22; App. I.7, Table 23, p. 38).
  • Behavioural Sensitivity under Physical Perturbations: Across matched counterfactual pairs where only the delivered physical reading moves out of band while digital and enterprise context stay fixed (Sec. 5.1, p. 10; App. I.7, Table 25, p. 40), frontier models flip reliably to ESCALATE (ΔP(ESCALATE) = +0.43 to +0.90, exact McNemar p ≤ 3 × 10⁻⁸), whereas withholding physical evidence entirely exposes high rates of ungrounded commitment in models that rely on digital priors (up to 47.1% for DeepSeek V4 Pro; App. I.7, Table 24, p. 39).

Q1 (Part 1): Credit for ESCALATE vs. ABORT under Withheld Evidence

Q1 (Part 1): "When a matched evidence regime removes the physical observation required to justify a COMMIT decision, should an agent that appropriately chooses ESCALATE or ABORT receive credit?"

3. Observed Behavior and Credit Policy under Withheld Evidence

  • ESCALATE receives full credit as justified caution: When a regime withholds a physical reading (or a requested reading fails/is occluded), our deterministic policy sets A_I = {ESCALATE} (Sec. 3.2, p. 6; App. B.1, p. 21). On the 255 withheld-reading COMMIT-gold worlds per model (Sec. 5.1, pp. 9–10; App. I.7, Table 24, p. 39), GPT-6-Astra (100%, 255/255) and Claude Opus 5.5 (97.6%, 249/255) appropriately choose ESCALATE and receive full credit under IJA.
  • ABORT is credited only when authorized by safety or battery conditions: Under our operational contract (Sec. 3.2, p. 6; App. B.1, p. 21), ABORT is reserved for explicit safety-gate refusals or battery < 20% conditions (tested in Family C; Table 3, p. 6), whereas missing diagnostic evidence requires escalating to a human operator (ESCALATE) rather than abandoning the workflow. Because no safety or battery condition occurred in any scored Family A episode (0/14,680), unprompted ABORTs under DIGITAL_ONLY (notably Gemini 3.1 Pro's 167/255 [65.5%] ABORTs after failed tool calls) are reported transparently in a separate "Uncredited ABORT" column (App. I.7, Table 24, p. 39) rather than conflated with justified ESCALATE.
AuthorsOpenReview ↗

Response to Reviewer o43z -3/3

W3 & Q2. Tool Discovery without Schemas and read_gauge Return Format

[W3 & Q2] Potential confound from names-only tool interfaces; call-failure audit and comparison with complete tool specifications; full agent-visible interaction and read_gauge return payload.

1. Tool Invocation Discovery and Full Call-Lifecycle Audit (W3)

W3 (Part 1): "Section 4 states that agents receive tool names without schemas or argument specifications, but does not clearly explain how valid tool invocations are discovered. Since failed calls cannot provide evidence and therefore reduce grounded success, some failures may reflect interface ambiguity..."

In Sec. 4 (pp. 7–8) and App. A.1–A.6 (pp. 17–20), we added a condensed interface specification and a call-lifecycle audit (Tables 4–5, pp. 19–20). In our two-call reference harness, a valid invocation requires only selecting a tool name from the regime-masked list offered in Turn 1 (25 in FULL, 23 in PHYSICAL_ONLY, 22 in DIGITAL_ONLY): {"tool_calls": [{"tool": "<name>", "args": {}}]}. Because the executor automatically binds every requested tool to the episode's asset_id and default parameters, argument-syntax or schema-formatting errors cannot occur by design.

Auditing all 14,440 traced episodes (74,464 tool requests) confirms that interface errors do not drive grounding failures:

  • 0 calls failed on arguments or JSON formatting; only 1 of 74,464 requests named an unknown tool; and 0 of 26,612 executed calls to gold-determining tools (read_gauge, capture_image, get_work_order, read_iot) failed (Sec. 4, p. 8; App. A.6, Table 5, p. 20).
  • All 11,528 failed calls were environment-side by scenario design (9,405 modality-absent calls where a sensor was not installed on that asset; 2,123 nested perception service errors).
  • The only harness mechanism that withheld requested evidence was the silent 8-call cap (78 of 11,940 V1 episodes [0.65%]; Sec. 4, p. 8), which we report explicitly and retain in the denominator as unresolved.

2. Comparison with Complete Tool Specifications (W3)

W3 (Part 2): "A comparison with complete tool specifications would help clarify what the benchmark actually measures."

We released the full schemas for all 67 tools across the 7 FastMCP servers in interface/tool_schemas/ in our anonymous code and in App. A.5 (p. 19). Because verbose descriptions can still influence tool selection, we state this confound in App. A.7 (p. 20) and ran a controlled comparison across 100 matched FULL Family A worlds on all five models (1,000 episodes; App. A.7, Table 6, p. 20):

  • All 3,792 argument sets supplied under the full-schema interface were valid, confirming call syntax is not a barrier.
  • Rather than raising grounded success, full descriptions lowered GSR from 73.4% to 23.0% (Δ = −50.4 pp, 95% CI [−56.8, −44.0]) and TDA from 73.6% to 49.0% (Δ = −24.6 pp, [−30.4, −18.6]), while the names-only arm reproduced our stored runs (GSR 76/70/69/77/75 vs. 76/73/71/77/80).
  • This drop reflects scope separation rather than syntax errors: the raw FastMCP descriptions include physical setup steps (power on, navigate, open panel) tested in Family C, causing models in Family A's single-round (8-call) harness to spend their call/token budget on setup before calling read_gauge. Thus, the names-only abstraction prevents setup overhead from confounding evidence-grounding scores in Family A, while full MCP schemas remain supported for multi-turn runners (App. A.5, A.7, pp. 19–20).

3. Complete Agent-Visible Interaction & read_gauge Return Format (Q2)

Q2: "Could the authors provide a complete agent-visible interaction specifying the initial input, tool signatures, returned observations, and required output format? In particular, does read_gauge return only an image, or does it also expose a numerical reading or other interpreted information?"

Yes. We now provide a complete, verbatim recorded episode in interface/example_episode/TRANSCRIPT.md in code and in App. A.4 (Listing 3, p. 19), alongside the initial system/user prompts (Listings 1–2, pp. 18–19) and all returned fields per tool (App. A.3, p. 18). We also updated Sec. 4 (p. 7) and Sec. 5 (p. 8) to explicitly state every route to the physical value:

  • read_gauge returns no image; it returns a structured payload containing a noisy numeric reading, confidence score, occlusion flag, operating range, and a text message restating the reading (Sec. 4, p. 7; Sec. 5, p. 8; App. A.3–A.4, p. 19).
  • capture_image attaches the PNG dial rendered at the noise-free value (Sec. 4, p. 7; Sec. 5, p. 8; App. A.3, p. 18).
  • get_work_order additionally exposes history_mean (a summary of past readings).

If these answers resolve your concerns, we would be grateful if you could reflect them in your assessment.

Reviewer_Qe8TOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a benchmark named InspectionBench, which has a focus on agentic physical AI in industrial inspection. The benchmark comprises 4120 episodes generated from different grounded worlds, scenarios and asset types. The motivation for this benchmarks is based in the automation of evidence collection for operational decisions through mobile robots, where agents receive alerts and asset context, with the options to perform certain actions via MCP tools and then form an operational decision. The authors identify six relevant benchmark capability criteria in the related work section of which only InspectionBench covers all of them, partially by building on top of existing benchmarks. Based on these criteria the authors describe the benchmark generation and later on the test of several agents in that environment. The evaluation of five frontier multimodal models shows that terminal decision accuracy can be much higher than evience grounded success, highlighting that correct operational decision do not necessarily imply successfull (complete) evidence acquisition.

Critical strengths
  • The paper adresses an important gap in the evaluation of agents. A core benefit within the overall contribution is the ability to distinguish between the terminal decision accuracy and the grounded successs rate along with the availability of all six benchmark attributes as defined in the related work section. The world to scenario compiler allows for controlled comparisons across evidence availability which enables ablation studies.
  • The paper contains an empirical study on the benchmark, that shows discrepancies between terminal accuracy and grounded success.
Critical weaknesses
  • The grounded success metric assures that required evidence was successfully acquired, but does not establish whether an agent actually used that evidence for the final decision. This limits the extent to which the benchmark might support conclusions about evidence-grounded reasoning rather than acquisition compliance.
  • The benchmark is grounded in real world images and expert supervision of data annotations and process decisions, but with only 116 facility collected images of very specific types of assets, the representativeness of the benchmark can be questioned. Although comprising thousands of episodes, the benchmark might still be limited to a very small underlying distribution. Also the asset-identity agreement and the gauge-value agreement within the automated data annotation pipeline are conceerning with high disagreements between the Gemini and GPT models. This might have an impact on their suitability for automated evaluation.
  • The escalate basline achieves 87.5% accuracy on the tested subset, which might imply a substantial imbalance in at least the data subset. It remains unclear how this affects the reported aggregate results and model rankings. It would be beneficial to show TDA and GSR ratios individually per gold action class (commit, escalate, abort). This would also give more insight into the model rankings.
Questions for authors
  • Given that the evidence delivery firewall does not gurantee whether agents actually reason from the acquired evidence, to what extent is GSR a valid measure of evidence grounding in decision making instead of terminal correctness combined with evidence acquisition?
  • How does performance vary across gold action classes and does action imbalance influence TDA and GSR rankings?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Qe8T -1/3

Thank you for your thoughtful review. Please find our point-by-point response across three comments below.

W1 & Q1. Evidence Delivery vs. Evidence Use, and What GSR Measures

[W1] "The grounded success metric assures that required evidence was successfully acquired, but does not establish whether an agent actually used that evidence for the final decision. This limits the extent to which the benchmark might support conclusions about evidence-grounded reasoning rather than acquisition compliance." [Q1] "Given that the evidence delivery firewall does not guarantee whether agents actually reason from the acquired evidence, to what extent is GSR a valid measure of evidence grounding in decision making instead of terminal correctness combined with evidence acquisition?"

We agree that GSR certifies evidence acquisition and delivery rather than internal causal reasoning (Sec. 4, p. 8; Sec. 5.1, pp. 9–10). By design, GSR serves as the necessary lower floor for evidence-grounded execution while GSR = 1 does not by itself prove causal reasoning, GSR = 0 when TDA = 1 proves that the agent committed without receiving the required physical evidence. Keeping GSR unchanged as Layer 1, we now report four complementary measurements alongside it in Sec. 3.2 (p. 6), Sec. 5.1 (pp. 9–10), and App. B.1, I.7–I.8 (pp. 21–22, 34, 38–40):

  1. Behavioural sensitivity to the delivered reading (Sec. 5.1, pp. 9–10; App. I.7, Table 25, p. 40): Across 100 matched world pairs per model and regime where only the delivered physical reading moves out of band while IoT, work order, and technician context stay fixed (921/1,000 complete pairs; 79 pairs with unparsed outputs excluded rather than counted as no-flips), P(ESCALATE) rises in every model and regime: Δ = +0.43 to +0.90 (exact McNemar p ≤ 3 × 10⁻⁸ in every cell; 92.9%–100% flip rate among original COMMITs; only 1 reverse flip in 921 pairs).
  2. Information-conditioned action admissibility (Sec. 3.2, p. 6; Sec. 5.1, pp. 9–10; App. B.1, pp. 21–22; App. I.7, Tables 23–24, pp. 38–39): Scoring each episode against the deterministic action set A_I justified by actually delivered observations shows that when DIGITAL_ONLY withholds the physical reading on 255 COMMIT-gold worlds per model, GPT-6-Astra (100%, 255/255) and Claude Opus 5.5 (97.6%, 249/255) choose justified caution (ESCALATE), whereas DeepSeek V4 Pro (47.1%, 120/255) and Qwen3.5 (32.5%, 83/255) make unsupported commitments. Pairing this with each model's matched-world change in COMMIT rate when the physical reading is available (FULL minus DIGITAL_ONLY: +0.36 Claude, +0.44 GPT, +0.73 Gemini, +0.43 Qwen vs. +0.02 [-0.02, +0.05] DeepSeek; Table 24, p. 39) separates evidence-dependent caution from default escalation, showing that DeepSeek's commitments follow IoT telemetry rather than physical verification. Requiring the terminal action to lie in A_I (Strict GSR, Table 23, p. 38) holds Claude (50.4% to 49.5%) and GPT (51.3% to 50.9%) steady, but drops Gemini (53.6% to 35.5%) and Qwen (52.1% to 35.9%) due to omitted work-order or technician checks.
  3. Explicit and validated observation attribution (EORR & VEA; Sec. 5.1, p. 9; App. B.1, Table 7, p. 21; App. I.7, p. 34): In our citation-enabled protocol (500 episodes/model), 97.5%–100% of GSR-successful episodes explicitly cite a delivered, asset-matched physical observation ID (EORR). Requiring the decision to be consistent with cited structured observations (VEA) retains 255 / 325 / 314 / 365 / 369 of 500 episodes (-5, -2, -12, 0, -9 below GSR), while exposing that 1.6%–11.8% of decisions still contradict the agent's own cited evidence.
  4. Claim-level entailment of explanations (App. I.8, p. 34; Table 26, p. 40): Across 30,912 extracted factual claims, we evaluate whether statements in the agent's rationale are entailed solely by delivered observations using deterministic, DeBERTa-v3-large NLI, and hybrid verifiers alongside cross-family LLM-judge audits (GPT-6-Astra and Kimi K3.1 Pro: 56%–57% [0.42, 0.71] fully supported on the Horvitz–Thompson weighted random stratum, n = 75; 35%–37% beyond the evidence; 0%–1% contradicted; inter-judge κ = 0.84).

As stated explicitly in Sec. 5.1, GSR measures correctness conditioned on evidence delivery, while Layers (1)–(4) measure behavioural sensitivity, admissibility, explicit attribution, and claim support without claiming to certify internal causal reasoning.

AuthorsOpenReview ↗

Response to Reviewer Qe8T -2/3

W2. Facility Image Representativeness and Annotation Agreement

[W2 Part 1] "The benchmark is grounded in real world images and expert supervision of data annotations and process decisions, but with only 116 facility collected images of very specific types of assets, the representativeness of the benchmark can be questioned. Although comprising thousands of episodes, the benchmark might still be limited to a very small underlying distribution."

We agree that the 116 facility photographs alone cannot represent the full diversity of industrial assets and inspection environments. We have clarified both the scope of this real-world sample and the complementary sources used to construct the benchmark in Sec. 3.1 (pp. 4–5) and App. C–E, H, J (pp. 24–26, 30, 34, 41):

  1. Representativeness of the facility data: The 116 photographs were collected at one facility during a single on-site session with a domain expert across straight-on, 15°, 30°, and oblique viewpoints (75 pressure/temperature gauges, 37 standalone temperature instruments, and 4 dyed-coolant spill proxies across four grounded assets; Sec. 3.1, p. 4). As stated explicitly in Sec. 3.1 (p. 4), the facility had no flow, vibration, or large-format gauges, and we do not interpret the thousands of compiled episodes as thousands of independent real-world photographs.
  2. How the visual and operational substrates are combined: The purpose of the 116 photographs is to ground visual appearance, viewing angles, and multi-instrument panel layouts in actual industrial imagery. They seed our PerceptionGauge pipeline alongside external gauge pools (SyncG: 1,567 images; DialBench/RPM-10K: 91 images) across a 49-row seed matrix (15 asset classes, 6 perception categories) to generate 1,296 controlled rendered scenes (856 readable, 440 unreadable) that systematically vary degradation, occlusion, scale, and glare (Sec. 3.1, p. 4; App. H, Table 16, p. 30). Complementing this visual layer, the benchmark incorporates 24,601 replayed operational records across six modalities (14,313 IoT, 9,757 acoustic from MIMII, 515 thermal, 7 access geometry, 6 RGB defect, and 3 work-order history records; Sec. 3.1, p. 4; App. J, Table 27, p. 41) plus chronological compressor replay trajectories (MetroPT-3; Sec. 5.1, p. 9; App. N, pp. 39–40).

[W2 Part 2] "Also the asset-identity agreement and the gauge-value agreement within the automated data annotation pipeline are concerning with high disagreements between the Gemini and GPT models. This might have an impact on their suitability for automated evaluation."

  1. Clarification and multi-VLM re-evaluation of annotation agreement (Sec. 3.1, p. 5; App. C–D, Table 9, pp. 24–25): The originally cited low agreement values (κ = 0.14 and κ = 0.18) compared two catalogues on 20 conflict-selected images using exact string matching rather than inter-model reliability across the 116 facility sequences. Following the protocol in IndustryBench §4.2, we independently re-annotated all 116 facility image sequences using three current-generation frontier VLMs (Claude Opus 5.5, GPT-6-Astra, and Gemini 4 Argon) under one frozen prompt and JSON schema (348 requests; 88, 116, and 110 schema-valid; App. C, p. 24). Across the three models, Krippendorff's α is 0.91 for coarse-mapped instrument type (76/83 three-way agreement) and 0.91 for normalized measurement unit (68/72), and 0.42–0.56 for category, recommended action, and readability (Table 9, p. 25). For continuous gauge values, interval α is 0.48 (95% CI [0.19, 0.98]); conditional on selecting the same unit and printed scale on dual-scale (psi/kPa, °F/°C) dials, the median pairwise difference is 1 scale unit, with 65.4%–80.0% of pairs within ±2% of full scale span (App. D, p. 25).
  2. Independent human practitioner audit (Sec. 3.1, p. 5; App. E, pp. 25–26): Four instrumentation and control practitioners (10 years combined experience, 120 person-hours) authored and adjudicated the benchmark. On an independent, blinded audit of 30 stratified scenarios across four ordinal dimensions (operational realism, difficulty calibration, evidence sufficiency, and gold-action correctness), three reviewers achieved Krippendorff's α = 0.84 (95% CI [0.76, 0.91]; 24 accepted, 5 revised, 1 rejected; App. E, p. 26).
  3. Zero impact on compiled-world evaluation scores (Sec. 3.2, pp. 5–6): Crucially, compiled-world ground truth and TDA/GSR are determined deterministically by sample_world(seed, cell), evidence availability, and operational requirements rather than VLM labels on the 116 photographs. Consequently, VLM annotation variance shifts compiled-world TDA and GSR by 0.0 pp and leaves all model rankings 100% unchanged.
AuthorsOpenReview ↗

Response to Reviewer Qe8T -3/3

W3 & Q2. Class Imbalance, Per-Class TDA/GSR, and Effect on Model Rankings

[W3] "The escalate baseline achieves 87.5% accuracy on the tested subset, which might imply a substantial imbalance in at least the data subset. It remains unclear how this affects the reported aggregate results and model rankings. It would be beneficial to show TDA and GSR ratios individually per gold action class (commit, escalate, abort). This would also give more insight into the model rankings." [Q2] "How does performance vary across gold action classes and does action imbalance influence TDA and GSR rankings?"

Thank you for raising this point. The 87.5% figure is the terminal accuracy of a constant always-ESCALATE policy on the 48 Family A episodes of the submitted panel, where 42 episodes have ESCALATE gold and 6 have COMMIT gold (ABORT is never gold in Family A; it is gold only in Family C on 28/112 episodes). That constant policy achieves 0% GSR and a balanced accuracy (BA) of 50.0%, compared with 54.8%–70.2% BA on the submitted panel and 61.3%–68.8% BA on the expanded pool for the five models. We now report TDA, GSR, and balanced accuracy separately per gold action class in App. I.6 (Table 22, p. 37) alongside our fixed reference policies (App. B.6, p. 24):

  1. Class imbalance inflates aggregate TDA for models that escalate by default: On the submitted panel (48 episodes), Claude Opus 5.5 and GPT-6-Astra are correct on 36/42 and 34/42 ESCALATE-gold episodes, but on only 2/6 and 3/6 COMMIT-gold episodes. On the larger traced pool (2,388 episodes per model: 765 COMMIT [32%], 1,623 ESCALATE [68%]; Table 22, p. 37), their COMMIT TDA is 29.0% and 35.4% versus 94.1% and 92.2% for ESCALATE (compared with 59.9% and 60.3% COMMIT TDA for Gemini 3.1 Pro and Qwen3.5).
  2. Model rankings depend on the metric: By balanced accuracy, Qwen3.5-397B ranks first (70.2% on the submitted panel; 68.8% on the expanded pool), whereas Claude Opus 5.5 ranks fourth on both (59.5% and 61.6%; paired world-cluster bootstrap 95% CI for Qwen minus Claude BA: +0.073 [0.054, 0.091]; App. I.6, pp. 33–34, 37). Similarly, on the expanded pool, Gemini 3.1 Pro ranks last on aggregate TDA (61.8%) but first on GSR (53.6%), with paired 95% CIs excluding zero (Gemini minus Claude: TDA −0.115 [−0.142, −0.087], GSR +0.033 [0.013, 0.052]). Throughout the paper, we state ranking differences only where paired world-cluster bootstrap 95% CIs exclude zero (App. K.5, p. 36).
  3. The TDA–GSR gap is driven by ungrounded decisions under DIGITAL_ONLY, not class mix: Within FULL and PHYSICAL_ONLY, per-class TDA and GSR coincide exactly for every model on the submitted panel, and on the 2,388-episode expanded pool they differ by at most 47 of 1,082 ESCALATE episodes (DeepSeek) and 0 COMMIT episodes (App. I.6, pp. 33–34). Thus, the TDA–GSR gap reflects correct terminal decisions made without physical evidence under DIGITAL_ONLY across both action classes, rather than an artifact of the action-class distribution.
AuthorsOpenReview ↗

A short question before the discussion closes

Dear Reviewer Qe8T,

Thank you again for your careful, positive review; your suggestions shaped most of the revision. As the discussion closes on November 18 (AoE), may we ask whether you have any remaining concerns? Even a brief note would be very welcome, and we would answer it right away. If the revision has addressed your concerns, we would be grateful if you could reflect this in your final assessment.

Thank you for your time and support.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to all reviewers

We thank all three reviewers (Qe8T, o43z, Mgp9) for their constructive feedback and address four shared concerns below, citing the revised PDF and anonymous repo (REPRODUCE.md).

1. Evidence delivery (GSR) vs. evidence use and justified caution (Qe8T W1/Q1, o43z W2/Q1, Mgp9 W1/Q2)

Qe8T Q1 & Mgp9 W1/Q2: "GSR assures required evidence was acquired, but does not establish whether an agent used that evidence... How often do episodes cite observations?" o43z Q1: "When a matched regime removes the physical observation... how does evaluation distinguish justified caution from an unsupported commitment?"

We agree that GSR certifies evidence delivery rather than internal reasoning (GSR = 0 with TDA = 1 proves an ungrounded action). Prompted by the reviewers, we strengthened our analysis of the TDA–GSR gap by introducing information-conditioned admissibility (A_I, IJA, and Strict GSR), explicit/validated observation citation checks (EORR and VEA), and counterfactual physical-reading perturbations. Together, these additions cleanly separate justified caution (ESCALATE when physical evidence is withheld) from unsupported commitments on digital priors, confirm that frontier models flip reliably when delivered readings move out of band (p ≤ 3 × 10⁻⁸), and verify explicit observation attribution across traces (Sec. 3.2, p. 6; Sec. 5.1, pp. 9–10; App. B.1, I.7–I.8, Tables 7, 23–26, pp. 21, 38–40).

2. Multi-VLM annotation reliability, human audit, and longitudinal grounding (Qe8T W2, o43z W1, Mgp9 W1)

o43z W1 & Qe8T W2: "With 116 facility images, representativeness can be questioned... low agreement in automated annotation may affect evaluation reliability." Mgp9 W1: "...the bench seems to be only for an inspection application evaluation rather than a solid agent capability test."

We clarified that the 116 facility images ground viewpoint/panel geometry and seed PerceptionGauge alongside external pools to generate 1,296 controlled scenes paired with 24,601 operational records (Sec. 3.1, pp. 4–5; Table 27, p. 41). Re-annotating all 116 sequences across three frontier VLMs under a frozen schema yields high agreement on instrument type and units (α = 0.91; Table 9, p. 25), complemented by a 3-practitioner blinded audit achieving α = 0.84 (App. E, pp. 25–26). Crucially, because compiled-world gold is computed deterministically from sample_world(seed, cell), VLM annotation variance shifts compiled-world TDA/GSR by 0.0 pp. Furthermore, chronological failure replays on MetroPT-3 / FutureSim show why Capabilities A–E are required to suppress false telemetry alarms while preventing upward failure-forecast drift (App. N, Figs. 14–15, p. 45).

3. Tool-interface specification, full-schema ablation, and reproducibility (o43z W3/Q2, Mgp9 W2/Q1/Q3)

o43z W3/Q2: "A comparison with complete tool specifications would clarify what the benchmark measures... does read_gauge expose a numerical reading?" Mgp9 W2/Q1/Q3: "What is the detailed agent harness... empirical controls, and episode accounting across constructed, executed, and scored episodes?"

We expanded Sec. 4 (pp. 7–8), App. A (pp. 17–20; Fig. 8, Listings 1–3, Tables 4–6), and App. K–L (Tables 30–34, pp. 42–43) to specify the two-call harness, exact read_gauge payload (numeric reading, confidence, occlusion flag; capture_image returns the PNG dial), complete episode accounting (4,071 scored episodes/model with 0 infrastructure failures), and a call-lifecycle audit confirming 0 syntax/argument errors across 74,464 calls. A new 1,000-episode ablation comparing names-only against full FastMCP schemas (Table 6, p. 20) confirms 100% valid arguments under full schemas while showing that verbose descriptions divert Family A's 8-call budget to setup steps tested in Family C. REPRODUCE.md provides one-command scripts (compare_runs.py) for API-free CPU rescoring (p. 11).

4. Resource-aware temporal grounding and per-class rankings (Qe8T W3/Q2, Mgp9 Q2)

Mgp9 Q2: "How does temporal grounding distinguish reliance on outdated observations from legitimate decisions not to re-observe under resource constraints?" Qe8T W3/Q2: "How does performance vary across gold action classes and does action imbalance influence TDA and GSR rankings?"

We thank the reviewers for raising these points, which led to two new trace-level analyses. Auditing all 4,690 repeated E visits shows zero stale reliance when budget was available, and we added Resource-Aware TVR (TVR^RA) to separate legitimate budget-exhausted safe abstentions (ESCALATE/ABORT) from ungrounded commitments on outdated readings, shifting repeated-episode TVR to 97.4%–100.0% for four models vs. 49.6% for the model prone to stale commitments (Sec. 5.3, p. 10; App. B.5, Table 8, p. 24). Likewise, reporting per-gold-class TDA/GSR and balanced accuracy across the 2,388-episode pool separates evidence-grounded capability from default escalation (App. I.6, Table 22, p. 37).

Area_Chair_5fMFOpenReview ↗

Reviewers-Authors Discussion Reminder

Dear Reviewers,

The review process for the paper has reached the rebuttal stage, and the authors have submitted their responses. Reviewers are encouraged to engage actively with these responses. Please clarify whether your initial concerns have been addressed and indicate if any new questions have arisen. Your meaningful participation is essential to maintaining a healthy ecosystem for ICLR.

Your AC.

AuthorsOpenReview ↗

Closing remarks from the authors

Dear Area Chair and Reviewers,

We thank all three reviewers for their constructive feedback, which shaped most of the revision. Reviewer Mgp9 confirmed that the revision addresses almost all of their concerns and raised the score. Reviewer o43z confirmed that the revisions address several original concerns and kept two open points: novelty and practical impact. We summarize the final state of the paper below.

What InspectionBench contributes. Existing benchmarks evaluate either static perception (SyncG, MMAD, FactoryBench) or digital enterprise workflows (τ-bench, AssetOpsBench, PHMForge), and none of them dispatches a physical agent to verify an alert before an operational decision (Sec. 1-2, Table 1). InspectionBench closes this physical-to-operational loop with 4,120 canonical episodes from 1,552 grounded worlds across six asset types. Its world-to-scenario compiler holds the world and gold decision fixed while it varies evidence access, so every gap between regimes isolates one cause. The results expose failures that terminal accuracy hides: correct-action rates exceed evidence-supported success by 23.2 to 29.8 pp for four of five models, and agents commit without physical evidence in up to 47.1% of evidence-withheld episodes.

How the revision addresses each review.

  • Qe8T and Mgp9 (GSR measures delivery, not use): four new layers test whether agents use the evidence. These are explicit observation citation (97.5% to 100% of GSR-successful episodes), validated attribution, information-conditioned admissibility, and matched reading perturbations that flip decisions in every model (exact McNemar p ≤ 3 × 10⁻⁸).
  • Qe8T and o43z (annotation reliability): three current VLMs agree at α = 0.91 on instrument type and unit, a blinded three-practitioner audit reaches α = 0.84, and annotation variance shifts compiled-world TDA and GSR by 0.0 pp.
  • Qe8T (class imbalance): App. I.6 reports TDA, GSR and balanced accuracy per gold class, and rankings change with the metric.
  • o43z and Mgp9 (interface and reproducibility): Sec. 4 and App. A specify the harness and the read_gauge payload, a 1,000-episode full-schema ablation and a 74,464-call audit rule out interface errors, and Table 31 accounts for every episode.

Practical impact and scope. A chronological replay of MetroPT-3 compressor data (76 false alarms, 11 true faults) shows that robot-acquired verification suppresses false escalations while it preserves fault visibility (App. N). Following Reviewer Mgp9, Section 6 now bounds our conclusions to an offline, single-round verification interface and leaves live multi-turn deployment to future work.

We will release the full facility data collection, and the anonymous repository already supports API-free rescoring through REPRODUCE.md. We thank the Area Chair and all reviewers again for their time.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.