acceptodds
Under review as a conference paper at ICLR 2027

FactoryBench: Evaluating Industrial Machine Understanding

Abstract

We introduce FactoryBench, a benchmark for evaluating time-series models and LLMs on machine understanding over industrial robotic telemetry. Q&A pairs are organized along four causal levels (state, intervention, counterfactual, decision) instantiating Pearl's ladder of causation, and span five answer formats: four structured formats are scored deterministically and free-form answers are scored by an LLM-as-judge voting protocol. We propose a scalable Q&A generation framework built around structured question templates, present FactoryWave (a dense, multitask, multivariate sensor dataset collected from a UR3 cobot and a KUKA KR10 industrial arm), and construct FactoryBench as a large-scale benchmark of over 70k Q&A items grounded in roughly 15k normalized episodes from FactoryWave, AURSAD, and voraus-AD. Zero-shot evaluation of six frontier LLMs shows that no model exceeds 50% (chance-corrected) on the structured levels and that the panel ordering reshuffles on the decision-making level, with the L1-L3 leader falling to the bottom of the L4 ranking; both findings reveal a wide gap between current models and operational machine understanding.We introduce FactoryBench, a benchmark for evaluating time-series models and LLMs on machine understanding over industrial robotic telemetry. Q&A pairs are organized along four causal levels (state, intervention, counterfactual, decision) instantiating Pearl's ladder of causation, and span five answer formats: four structured formats are scored deterministically and free-form answers are scored by an LLM-as-judge voting protocol. We propose a scalable Q&A generation framework built around structured question templates, present FactoryWave (a dense, multitask, multivariate sensor dataset collected from a UR3 cobot and a KUKA KR10 industrial arm), and construct FactoryBench as a large-scale benchmark of over 70k Q&A items grounded in roughly 15k normalized episodes from FactoryWave, AURSAD, and voraus-AD. Zero-shot evaluation of six frontier LLMs shows that no model exceeds 50% (chance-corrected) on the structured levels and that the panel ordering reshuffles on the decision-making level, with the L1-L3 leader falling to the bottom of the L4 ranking; both findings reveal a wide gap between current models and operational machine understanding.

Primary area:
datasets and benchmarks
Keywords:
benchmark, time series, large language models, causal reasoning, robotics, industrial telemetry, evaluation, datasets

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JzmZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This is a dataset and benchmark paper, and is designed for evaluating the LLM Agent.

Critical strengths

The paper proposes a well-defined, well-known problem. The direction and procedure defined in the paper will lead to creating more and more datasets, but these methods are trivial.

Critical weaknesses

I have not used an LLM or an AI Agent anywhere.

  • I carefully read the referenced work and found two major observations. First, thank you for ensuring the paper references are well formatted. Out of curiosity, I see two distributions: a) many references referring to the time series foundation models (which is good); b) various models provided. However, I noticed the paper is submitted to the Dataset and Benchmark tracks, and the title is “Industrial Machine”; I think the author should quickly check which sets of papers are available for this domain. For example, IndustryBench, FailureSensorIQ, AssetOpsBench, IndustryEQA, etc. This suggests the paper should quickly explore similar Benches in the Same domain.

  • I found some published papers, but they are cited as arXiv reports; it may be better to revise those citations.

  • I printed the paper, and Figure 1 is not readable. Please ensure it is readable and choose a better color combination.

  • The task classification in the introduction section is well defined. I appreciate it. You could connect it to industry-related tasks, such as IndustryBench.

  • Another point the paper mentions is the Scalable Question Generation Framework, and I recall pioneering work like Self-Instruct, so I will look into more technical innovations in these scalable frameworks. The statement in lines 122-124 may not be technically correct. AssetOpsBench and IndustryAssetEQA have covered some aspects of LLMs’ reasoning on time series data via foundation models. Please cross-check and adjust.

  • Questions are derived by following standard procedures such as Pearl’s Ladder. I recall that ITFormer also prepared MCQA-style questions. Please include that as well. The paper provides a long explanation of four types, but it can be used effectively to include examples in the main paper. Section 3.2 needs to be more technical; if there is any innovation, provide a reusable script to regenerate more data. Include dataset statistics so we can understand the scale.

  • Section 4.1, the main part, does not fully explain the context-free grammar (developed by the student/PhD). Key components of any dataset and benchmark paper should be in the main copy. Please See How IndustryEQA has written the key components. In Line 214, the paper mentions “Each of the 21 templates…”… I could not find where it is mentioned that a total of 21 templates were developed. Lines 211-213 point to Appendix Sections D, E, and F. The key point must be brought back to the main text; this is the art of writing an effective, self-contained paper. It doesn't tell me how I would implement the methodology if I read, say, Section 3.1, Section 3.2, and Section 3.3.

  • Long paragraphs on page 4 and page 5 need attention. Flow is an issue. For every section, give an outline of what should be discussed or expected at that point in time. For example, look at sections 4.1, 4.2, and 4.3; the headers don't show how these three are interconnected. Maybe there is a reference in Figure 1.

  • Line 263 mentions that a PhD-level expert created a dataset. This needs a study of human labeling error or bias. Section 4 doesn't justify its availability; is the use of CFG an innovation, or is it already used? Including one example of each task type in the main paper could have been helpful. This also raises another question: did the authors conduct a similarity search between their question and, say, questions from ITFormer or AssetOpsBench, and note any differences?

  • Section 5.1: again, many references to Appendinx. It was unclear how TSFM models answered Level 1 questions. My understanding is that the question goes to the LLM agent, and it may decide to use the TSFM model for any analysis. Is my understanding clear? In a typical dataset paper, after preparing the DB, you show some statistics, like the number of options. Especially in MCQA, how do you ensure your distractors are sampled appropriately? I found the DiagnosticIQ paper that describes the pipeline well and provides a detailed dataset distribution of options, etc. I'm currently reading the research paper on page 6, and it’s not very clear.

  • Section 5,2, First result. I haven't seen anything: no leaderboard, no results, and the conclusion is highlighted in bold. Line 295-296. Please discuss the parameters that are used for the LLM. Then walk us through the overall result, and then give some insights. For 5.2, I need an example to understand the different failure modes.

  • The fonts in Figure 2 (b) are not easily readable. The purpose of Factory vs. Robot needs to be clearly articulated. IndustryEQA uses Factory as an environment to create Q&A and solve problems; however, here the base-row data comes from Robot. The model cites many time series models; however, the main paper includes no in-depth analysis.

  • The paper abstract includes a link to a Hugging Face repo that, in my opinion, lists the IDs or names of people associated with HF repositories. I will ask the area chair to investigate whether this violated the double-blind policy. Kaggle provides a clean anonymous way of sharing the data. I highly recommend it for future submissions.

Questions for authors

I have read the paper and have no question that can change my assessment.

6 comments
AuthorsOpenReview ↗

Now self-contained, positioned against 8 benchmarks, with no data or question overlap

Thank you for the detailed review. Following it, Section 2 now positions FactoryBench against eight industrial and time-series QA benchmarks, the main text carries the grammar, one example per level and a leaderboard, and we show that FactoryBench shares neither data nor questions with EngineMT-QA or AssetOpsBench.

Related industrial benchmarks. Section 2 now discusses AssetOpsBench, IndustryAssetEQA, SenTSR-Bench, FailureSensorIQ, DiagnosticIQ, IndustryBench, IndustryEQA and EngineMT-QA (ITFormer). What sets FactoryBench apart is its method, not only its robots: Level 1–3 answers are computed from recorded multichannel telemetry with physically injected faults, counterfactual answers come from matched physical reruns, and the four levels run from reading the machine state to choosing a repair. We also corrected lines 122–124: AssetOpsBench and IndustryAssetEQA do evaluate LLM reasoning over industrial telemetry, and the sentence now states the actual difference.

Question generation and Self-Instruct. Our generator uses no LLM. Level 1–3 answers are computed from recorded signals and fault labels, and Level 4 references come from manufacturer documentation and expert protocols. We claim no novelty for the template grammar itself; the contribution is checkable labels from physical recordings.

Grammar, the 21 templates, appendix content. Section 4.1 now defines the grammar (template, typed slots, slot sources) with a worked derivation, states the 21 templates and summarises Appendices D–F.

Examples, statistics, regeneration, industry tasks. Table 1 shows one example per level, Section 4.1 carries the technical detail of generation, and Section 3.2 now points to a new statistics table covering episodes, robots, sampling rates, tasks, faults and items per level and format (Table 8, App. C). Section 4.1 names the regeneration entry points, and Section 3.1 maps the four levels to condition monitoring, impact assessment, what-if analysis and troubleshooting.

Figures. Figure 1 is redrawn at print size (labels 7 to 8.6 pt, up from 3.3 to 4.5 pt) and reads in greyscale; Figure 2b is redrawn for all six models (Figure 12).

Citations. Six references now cite the published version: PatchTST, Time-MQA, ITFormer, QuAnTS, Peyrard & West and TimeSeriesExam. TSAQA has no published version yet.

Human labelling. We added a blind check: two external robotics engineers score 0.75 and 0.70 raw on 30 rebuilt Level 3 items against 0.33 for the best model (Table 37). Level 1–3 labels are computed, not annotated; a released 102-item expert solution set is audited ground truth, and an independent solver blindly re-solving 20 Level 1–2 items reproduced 19 stored answers (App. G.2).

Overlap with ITFormer and AssetOpsBench. FactoryBench shares neither data nor questions with these benchmarks. EngineMT-QA is built on simulated turbofan engines and AssetOpsBench on chiller telemetry, whereas FactoryBench uses recorded industrial robot arms with injected faults. A similarity search confirms this for the questions themselves: across all 949 Lite question stems, the highest similarity to any of 14,747 EngineMT-QA and 440 AssetOpsBench questions is 0.14 in wording (TF-IDF) and 0.52 in meaning (MiniLM), against a median of 1.00 between EngineMT-QA's own test and train questions (App. Q.1, Table 35).

Section 5: leaderboard, decoding, forecasters, options. Section 5.2 now opens with a leaderboard (Table 2) that includes a trivial baseline per template, and Section 5.1 states the decoding (temperature 0, one message, 2,000-token cap). Section 5.1 now adds the time-series models TimesFM-2.5 and Chronos-Bolt as standalone forecasters, not LLM tools: they score level with Claude on 166 L1.7 items (+63.1 and +60.6 vs +61.4), and TimesFM-2.5 beats it on Lite L1.7, L2.4 and L2.5 (+37.9/+15.2/+24.9 vs +34.9/+3.0/+18.3; Table 22). Section 4.1 gives the number of options and how distractors are sampled, unlike DiagnosticIQ's embedding-based sampling, and Section 5.2 now walks from the overall result to the per-level findings with a failure example.

Factory and robot. Section 1 now explains the name: the data are telemetry from industrial robots working in factory cells, rather than a factory scene as in IndustryEQA.

Hugging Face link. We understand the concern, and to our knowledge the link does not identify us. The names on the page appear to be Hugging Face's list of users who liked the dataset; none of them is an author, and the dataset card and Croissant metadata name no author or affiliation. Kaggle is a good suggestion that avoids this issue, and we will use it for future anonymous releases.

Thank you: these suggestions made the paper self-contained and place it among the industrial benchmarks you named.

Reviewer_JzmZOpenReview ↗

Ack and More Feedback (on revised manuscript)

I acknowledge that I read the rebuttal response. I have a few pieces of feedback on the (revised version) of the paper

  • Kindly note that in the revised manuscript, we cannot say which parts are modified and which parts are original. Ideally, many authors use different colors to show the changes.
  • I always suggest not including a laundry list of the paper (lines 069-070). As said earlier, writing the paper requires a careful presentation of the related work instead of just listing it.
  • Font size in the table (in the main paper ): I expect the font size to be as close to the main text as possible. Readability improves. (Tables 1, 2, 3). Space is not an issue, 10th page is 50% empty.
  • Section 3 and Section 4 seem reformatted, but their quality/representation may need more work.
  • \cite vs \citep - pls prefer to use \citep,
  • Thanks for the revised manuscript; it has improved a lot based on the review.
AuthorsOpenReview ↗

Second revision: changes marked in colour, citation lists in the introduction improved, larger tables, shorter paragraphs

Thank you for reading the revision and for the further feedback. We adopted all of it.

Marked changes. The supplementary material now contains the paper with every change from the original submission marked: added text in blue, removed text struck through in red.

Citation lists in the introduction (lines 069–070). The list of forecasting architectures is now a sentence that says what each line of work adds: the Transformer adapted to long horizons and to patches of the series, pretrained zero-shot forecasters such as Chronos and TimesFM, and self-supervised event prediction. The list of time-series Q&A benchmarks in the next paragraph is rewritten the same way: TimeSeriesExam and TSAQA ask about univariate series; ChatTS, Time-MQA, MTBench and QuAnTS use multivariate series that are not recorded from a machine; and the closest, EngineMT-QA, uses simulated turbofan telemetry rather than a robot under closed-loop control.

Table fonts. Tables 1–3 font size is increased (9pt, up from 5 to 7pt). Table 2 is no longer scaled down, and its intervals sit under each value. The appendix tables are now set at 9 pt as well.

Sections 3 and 4. Following your original review, the headings now show how the parts connect: 3.2 "Signals and the setpoint–context–effort schema", 4.1 "Templates: from episode labels to questions and answers", 4.2 "Fault and counterfactual recordings in FactoryWave" and 4.3 "Level 4 references from manufacturer documentation". Section 3 now opens with an outline, as Section 4 does, and the two longest paragraphs of Section 4.1, of 16 and 25 lines, are each split in two.

Citations. All citations now use \citep.

Reviewer_ZMnjOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

FactoryBench evaluates industrial machine understanding through 21 question templates spanning state interpretation, intervention, counterfactual reasoning, and engineering decisions. It constructs 69,691 Q&A items from approximately 15,000 normalized robot-telemetry episodes across FactoryWave, AURSAD, and voraus-AD, with a balanced 3,000-item Lite evaluation set. Structured answers use deterministic, chance-corrected scoring; free-form decisions use the median of three LLM judges without chance correction. Counterfactual references approximate alternative outcomes using matched physical reruns, while troubleshooting references draw on manufacturer documentation and expert protocols. Six LLMs, forecasting baselines, expert checks, and a tool-augmented agent reveal substantial performance gaps, with limitations involving counterfactual validity, numeric-score calibration, and the role of vendor-specific knowledge.

Critical strengths
  1. The benchmark turns difficult-to-obtain industrial robot telemetry, including controlled fault injections and matched physical reruns, into a valuable question-answering resource. Although FactoryWave was previously released with FactoryNet, the integration, annotation, and conversion of these data and additional public datasets into reproducible evaluation items constitute a useful resource contribution. The four-level taxonomy and extensible evaluation pipeline provide a systematic organization of industrial machine-understanding tasks. The 21 templates, multiple answer formats, explicit scoring rules, and balanced Lite subset make the resource useful for studying distinct aspects of machine understanding, even where the causal interpretation requires further validation.

  2. The empirical analysis is broad and transparent, including human-expert checks, noise substitution, specialized forecasting baselines, a tool-augmented agent, inter-judge agreement, and failure analysis. The explicit discussion of approximate counterfactual references and vendor-knowledge confounds helps readers assess the scope and limitations of the reported findings.

Critical weaknesses
  1. The L2/L3 evaluation lacks controlled comparisons that distinguish causal reasoning from conditional prediction. Consequently, the benchmark’s construct validity as a measure of causal capabilities remains insufficiently established.

  2. The L4 ranking reversal conflates differences in fault identification, vendor-specific knowledge, and answer formats and scoring procedures. Without controlled experiments isolating these factors, the results do not sufficiently support the claimed dissociation between signal comprehension and engineering decision-making.

Questions for authors
  1. What evidence distinguishes causal reasoning on L2/L3 from conditional prediction or trajectory continuation? Please clarify whether matched-baseline tests that vary only the intervention, or baselines that ignore the intervention description, support the causal interpretation; otherwise, how should the claims be qualified?

  2. Does the L4 ranking reversal persist when fault identification, vendor-specific knowledge, and answer-format or scoring differences are controlled? For example, do comparisons supplying the correct fault label or relevant manual excerpt support a separation of capabilities, rather than differences in available knowledge or evaluation format?

3 comments
AuthorsOpenReview ↗

Both proposed controls run: Level 4 is limited by diagnosis, Level 2 renamed

Thank you for the constructive review. We ran both controls you proposed on all six models, and the fault-label control produced one of the clearest results of the revision: with the correct root cause, every model rises from under 20% to over 70% on Level 4, so decision-making there is limited by diagnosis. Scores are chance-corrected points with 95% bootstrap CIs; Level 4 is the median of three judges, in %.

W1 and Q1. Causal reasoning or conditional prediction on Levels 2 and 3? Conditional prediction, and the paper consistently adopts this. Removing, swapping or contradicting the stated event, with the rest of the prompt identical, significantly lowers no API model's score (758 items). The table gives, for each arm, the range of paired changes over the six models; of these 36 model × arm × level changes, the only significant drop is the −5.1 under Swapped on Rebuilt L3, from the smallest model, Qwen3-4B ([−9.5, −0.8]).

Paired Δ, range over 6 modelsRemovedSwappedContradicted
L2, 572 items−2.9 to +4.3−2.1 to +1.8−2.6 to +4.7
Rebuilt L3, 186 items−0.5 to +7.7−5.1 to +4.1−2.5 to +6.0

Following the rule we fixed before the run, Level 2 is renamed "event-conditioned prediction" (formerly "intervention"), the abstract no longer calls the levels causal, and Sec. 5.3 and 5.6 state that Levels 2 and 3 measure event-conditioned prediction.

Level 3 itself is now built from 58 physically matched pairs of fault-free and fault runs (244 items), with the reference taken from the fault run. We also tested it as a strict counterfactual query by adding the fault-free run's recorded continuation to the context. On the 67 items where the fault changes the answer, no API model beats the best trivial baseline (skill −30 to −51, every CI below zero; App. Q.2). In a blind check on 30 rebuilt items, by contrast, two external robotics engineers score 0.75 and 0.70 raw against 0.33 for the best model, and 0.60 on single-value forecasts against 0.10 for repeating the last value (App. Q.3).

W2 and Q2. Does the Level 4 reversal survive controls? No, and the controls explain Level 4 better: it is limited mainly by diagnosis. With the correct fault label every model rises from 5.2–18.7% (Original) to 73.3–89.9% (+ fault label), whereas a manual excerpt alone, which gives the manufacturer's recovery procedure among distractor procedures, reaches only 40.0–47.4% (+ manual excerpt).

Level 4, %Original+ fault label+ manual excerpt
Claude Sonnet 4.614.389.946.3
GPT-5.118.088.843.9
DeepSeek V3.216.473.340.0
Mistral Large 314.586.047.4
Qwen3-235B18.784.146.9
Qwen3-4B5.288.846.3

In the Original column the four API models lie within 3.7 points (14.3 to 18.0) with overlapping CIs, so the ranking difference is not significant; the abstract, Sec. 5.4 and the conclusion now state the diagnosis result instead. Because the label also supplies the root-cause half of the rubric, a judge outside the panel rescored the repair procedure alone: the four API models still gain 27 to 57 points, every CI above zero (Table 34). The scores hold with a fourth judge outside the panel and without each judge's own verdicts (Table 32); the four judges agree with Fleiss κ 0.68 over all arms and 0.48 on the original arm, where most scores are low (App. B.2).

Your two controls now give the paper a precise account of what Levels 2 and 3 measure and of why models fail Level 4.

Thank you for running the additional controls and clarifying the findings. The event-description experiments and the fault-label and manual-excerpt comparisons directly address my main concerns. I appreciate your willingness to revise the interpretation based on these results. Please make sure the more cautious claims are reflected consistently throughout the paper and submission page, including the abstract and TL;DR, particularly regarding causal reasoning and the claimed separation between signal comprehension and decision-making.

AuthorsOpenReview ↗

Cautious claims now consistent throughout; TL;DR and keywords at camera-ready

Thank you; we agree, and the second revision makes the wording consistent. Section 3.1 now describes Level 2 as predicting the consequences of a stated event, typically a fault, and Level 3 by its reference, a physical rerun with the fault injected. The phrases "Pearl's three causal levels" and "the dataset structure encodes causality" are removed, and the conclusion says the levels are organized by Pearl's ladder. The abstract, Section 5.4 and the conclusion state the diagnosis result in place of a separation between signal comprehension and decision-making.

OpenReview does not allow to edit the TL;DR and keywords during the discussion. At camera-ready we will replace the TL;DR with: "A four-level benchmark on industrial robot telemetry: frontier LLMs stay below 50% chance-corrected and fail at decision-making mainly by misdiagnosing the fault." We will also replace the keyword "causal reasoning" with "counterfactual prediction" and "fault diagnosis".

Reviewer_Ux3UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents FactoryBench, an industrial robotic telemetry benchmark extending Pearl’s causal ladder into four reasoning levels (state, intervention, counterfactual, decision). It contributes the FactoryWave dataset and 70k+ QA pairs. Evaluations of LLMs and time-series models reveal a major gap: strong causal performers fail at actionable engineering decisions.

Critical strengths

S1. This paper introduces FactoryBench to evaluate an important and underexplored problem: whether LLMs can understand dense multivariate telemetry from industrial robotic systems beyond conventional forecasting or anomaly detection.

S2. The four-level evaluation framework, covering state understanding, intervention, counterfactual reasoning, and decision making, provides a clear and systematic way to assess different levels of machine understanding.

S3. The benchmark is grounded in real robotic data from multiple platforms and contains 69,691 Q&A items generated from approximately 15k episodes, providing substantial scale and diversity for evaluation.

Critical weaknesses

W1. The benchmark mainly evaluates existing LLMs through question answering over pre-constructed industrial telemetry tasks. It remains unclear how well the benchmark reflects more realistic industrial settings where models need to continuously interpret raw sensor streams and make decisions under changing operating conditions.

W2. Although the benchmark contains a large number of Q&A items, many questions are generated through predefined templates and task construction procedures. This raises concerns about the diversity of the evaluation and whether performance on FactoryBench can generalize to unseen machines, failure modes, and industrial environments.

Questions for authors

Q1. How well does performance on FactoryBench generalize across different machines? For example, have the authors evaluated a setting where questions from one robot or dataset are held out during benchmark construction/evaluation and used exclusively for testing? This would help clarify whether the benchmark measures transferable machine understanding or reasoning tied to specific platforms and signal distributions.

Q2. The 69,691 Q&A items are generated from only 21 templates. Could the authors provide additional evidence that models cannot exploit recurring template structure? In particular, would model rankings remain similar under held-out templates or substantially rephrased questions?

Q3. For Level 3, the human expert achieves only 0.625 and does not outperform the model panel, while the paper also reports that the L3.5 non-LLM baseline obtains +37.2% despite its assumed chance level. How confident are the authors that the current Level-3 evaluation reliably measures counterfactual reasoning rather than limitations in the task construction or scoring procedure?

Q4. For Level 4, some recovery protocols are mapped from manufacturer documentation, while others are directly authored by robotics experts. How are the expert-authored protocols validated, and how sensitive are the Level-4 results to alternative valid troubleshooting procedures?

3 comments
AuthorsOpenReview ↗

Level 3 rebuilt from physical reruns: engineers score 0.75, the best model 0.33

Thank you for the careful review. Your Q3 led to the most important addition of the revision: we rebuilt Level 3 from physically matched reruns, and in a blind check two external robotics engineers score 0.75 and 0.70 raw against 0.33 for the best model. Scores are chance-corrected points (0 = chance) with 95% bootstrap CIs unless marked as raw scores or percentages.

W1. Continuous streams and changing conditions. FactoryBench deliberately tests fixed windows, and the revision states this scope. Each item is a 3.2 to 6.4 s window at 10 Hz, which makes every answer checkable against the recording. The tool-using agent in Appendix J is our most interactive setting; a closed-loop version is a natural next step.

W2 and Q1. Generalisation across machines. On Levels 1–2 the leading model is the same on every robot. We scored the four templates shared by all source datasets (L1.1, L1.7, L2.6, L2.7) separately per dataset: FactoryWave (UR3 and KUKA KR10), AURSAD (UR3e) and voraus-AD (Yu-Cobot). Claude Sonnet 4.6 is first on all three; the other three models change order (Kendall τ 0.33 to 0.67). voraus-AD is the hardest for each API model, and only Claude is above chance there (+7.8 [+0.6, +14.4]). No model is trained on FactoryBench, so every robot is unseen by every model (Sec. 5.3, Table 29). Level 3 covers one robot: 58 UR3 pairs with three collision faults, since the KUKA reruns showed no measurable fault effect (Sec. 4.2). Compound and gradual faults remain outside scope: the 27 injected faults are atomic and fast (Sec. 5.6).

Q2. Template exploitation. The controls show that the strongest model reads the signal rather than template patterns, while most models barely use it. Removing the telemetry costs Claude 13.8 points [5.7, 21.5] and no other model a significant amount (Table 28), which is the gap the benchmark is built to expose; replacing it with another episode's telemetry lowers every API model by 6.5 to 9.6 points. Paraphrased templates move scores by −8.4 to +3.1, and Claude stays first under every paraphrase and whichever of the 19 L1–3 templates is dropped. On the six templates where a baseline reading only the question and options beat chance in v1.0, it now scores −10 to −5, and every Level 1–3 template reports a trivial baseline score (Sec. 5.3, App. B.1, App. Q).

Q3. Does Level 3 measure counterfactual reasoning? The rebuilt Level 3 is a counterfactual test that experts solve and models do not: in the blind check the engineers score 0.75 and 0.70 against 0.33 for the best model, and on the 55 numeric items where the fault changes the answer the models score 0.07–0.14, around the level of repeating the last value (0.10 / 0.11). Each item comes from a matched pair of runs: the same program, pose, payload and controller, rerun with the fault injected at the recorded time. Before the injection the joint positions agree within 0.5° (median); after it the runs diverge by a median 29 times their earlier gap, against 1.2 at a placebo time, so the reference captures the fault's effect. The model sees the fault-free run up to the injection and must predict the run with the fault.

Rebuilt Level 3 (raw score)ModelsReference
Blind check, 30 itemsbest model 0.33engineers 0.75 / 0.70; trivial baseline 0.37
55 numeric items where the fault changes the answer0.07–0.14, all API modelsrepeating the last value 0.10 / 0.11

The +37.2 you noted came from a tolerance band that regression hits easily; v1.1 therefore measures each template's trivial baseline instead of assuming a chance level. The earlier expert score of 0.625 was on the items before the rebuild (Sec. 4.2, 5.2, Table 3; App. G.2).

Q4. Expert protocols and alternative procedures. Level 4 stays hard under every rubric we tried, and the diagnosis result holds. A judge outside the panel rescored all answers by root cause alone and with a reference-free rubric that credits any valid, safe procedure: every API model stays below 30%. Given the correct root-cause label, all models reach 73 to 90%, and scoring the repair procedure alone, the four API models still gain 27 to 57 points. The order among models shifts with the rubric (τ 0.00 and +0.33), so the paper reports Level 4 scores but draws no conclusion from their order. Independent validation of the expert protocols is stated as a limitation (Sec. 5.4, 5.6, App. Q).

Thanks to your questions, Level 3 now rests on matched physical reruns and an expert check, and the signal controls show what the scores measure; we see both as a clear strengthening of the benchmark.

AuthorsOpenReview ↗

Level 3: engineers score 0.75 and 0.70, the best model 0.33

Thank you again for your review. Your Q3 on Level 3 led to the largest change of the revision. Each Level 3 item now comes from a pair of physical runs: the same program, run once without and once with the fault. In a blind check on 30 of these items, two external robotics engineers score 0.75 and 0.70, and the best model 0.33 (Table 37). Our reply above also answers Q1, Q2 and Q4. Happy to answer anything else before the discussion closes tomorrow.

AuthorsOpenReview ↗

Q1–Q4: same leading model on every dataset; on Level 3, engineers 0.75 and 0.70, the best model 0.33

Your questions led to several of the main additions of the revision. Scored separately on FactoryWave, AURSAD and voraus-AD with their shared templates, the same model leads on all three (Q1), and it stays first under paraphrased templates and with any template left out (Q2). Level 3 now comes from pairs of physical runs, and in a blind check on 30 of its items two external robotics engineers score 0.75 and 0.70, the best model 0.33 (Q3). Under alternative Level 4 rubrics, including one that credits any valid, safe procedure, every API model stays below 30% (Q4). We hope this shows how much your review strengthened the paper, and would be glad if it is reflected in your final assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Eight new experiments, 30,000+ new model answers: same finding, stronger evidence

We thank the reviewers for careful and constructive reviews. In response we ran eight new experiments, together more than 30,000 new model answers, with every Level 4 answer scored by four judges. They confirm the paper's central finding, that frontier LLMs are far from reading industrial robot telemetry the way an engineer does, and they now show precisely where the models fail. The data sources and the four-level structure are unchanged.

New experiments and what they show

  • Fault-label control on Level 4: failures come from diagnosis. Given the correct root-cause label, every model rises from under 20% to over 70%; scoring only the repair procedure, the four API models still gain 27 to 57 points. An excerpt from the manufacturer's manual, with the right procedure among distractors, helps far less (ZMnj, Ux3U).
  • Event controls on Levels 2 and 3: Level 2 is renamed. Removing, swapping or contradicting the stated event does not significantly lower any API model's score (758 items). Level 2 is therefore now "event-conditioned prediction" (formerly "intervention"), and we no longer claim causal reasoning for Levels 2 and 3 (ZMnj).
  • Signal ablations and paraphrases: only the strongest model clearly reads the signal. Removing the telemetry costs Claude Sonnet 4.6 13.8 points and no other model a significant amount, though another episode's telemetry lowers every API model's score; Claude stays first under paraphrased templates and whichever template is dropped. That most models barely use the signal is the gap FactoryBench measures (Ux3U).
  • Scores per robot dataset: the same model leads on every robot on the Level 1–2 templates shared by FactoryWave, AURSAD and voraus-AD (Ux3U).
  • Leaderboard with a trivial baseline per template, and time-series foundation models on the same items: the strongest model clearly beats the baseline only on Level 2, and on forecasts TimesFM-2.5 matches or beats it (JzmZ).
  • Alternative Level 4 rubrics and an overlap search: Level 4 stays below 30% for every API model under each rubric, and no FactoryBench question resembles one from EngineMT-QA or AssetOpsBench (Ux3U, JzmZ).
  • Level 3 rebuilt from 58 physically matched pairs of fault-free and fault runs (244 items): where the fault changes the outcome, no API model clearly beats repeating the last observed value, even when shown how the fault-free run continued (Ux3U, ZMnj).
  • Blind expert check of Level 3: human experts score far higher than frontier models. Two external robotics engineers reach 0.75 and 0.70 raw; the best model reaches 0.33 (Ux3U, JzmZ).

FactoryBench v1.1. Appendix P documents every change: Level 3 is rebuilt from the matched reruns; templates whose option wording or channel names gave away the answer are fixed; and every Level 1–3 template reports a trivial baseline score, such as repeating the last value, with skill over it as the recommended metric. With Level 3 rebuilt, the pool shrinks from the 69,691 items cited in the reviews to 66,996.

Setup. Four API models (Claude Sonnet 4.6, GPT-5.1, DeepSeek V3.2, Mistral Large 3) and two open-weight models (Qwen3-235B, Qwen3-4B); one user message, temperature 0, 2,000-token cap. GPT-5.1 ran at API default sampling outside the leaderboard and event controls; where measured, this moves no mean score by more than 3 points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.