- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This is a dataset and benchmark paper, and is designed for evaluating the LLM Agent.
Critical strengths
The paper proposes a well-defined, well-known problem. The direction and procedure defined in the paper will lead to creating more and more datasets, but these methods are trivial.
Critical weaknesses
I have not used an LLM or an AI Agent anywhere.
-
I carefully read the referenced work and found two major observations. First, thank you for ensuring the paper references are well formatted. Out of curiosity, I see two distributions: a) many references referring to the time series foundation models (which is good); b) various models provided. However, I noticed the paper is submitted to the Dataset and Benchmark tracks, and the title is “Industrial Machine”; I think the author should quickly check which sets of papers are available for this domain. For example, IndustryBench, FailureSensorIQ, AssetOpsBench, IndustryEQA, etc. This suggests the paper should quickly explore similar Benches in the Same domain.
-
I found some published papers, but they are cited as arXiv reports; it may be better to revise those citations.
-
I printed the paper, and Figure 1 is not readable. Please ensure it is readable and choose a better color combination.
-
The task classification in the introduction section is well defined. I appreciate it. You could connect it to industry-related tasks, such as IndustryBench.
-
Another point the paper mentions is the Scalable Question Generation Framework, and I recall pioneering work like Self-Instruct, so I will look into more technical innovations in these scalable frameworks. The statement in lines 122-124 may not be technically correct. AssetOpsBench and IndustryAssetEQA have covered some aspects of LLMs’ reasoning on time series data via foundation models. Please cross-check and adjust.
-
Questions are derived by following standard procedures such as Pearl’s Ladder. I recall that ITFormer also prepared MCQA-style questions. Please include that as well. The paper provides a long explanation of four types, but it can be used effectively to include examples in the main paper. Section 3.2 needs to be more technical; if there is any innovation, provide a reusable script to regenerate more data. Include dataset statistics so we can understand the scale.
-
Section 4.1, the main part, does not fully explain the context-free grammar (developed by the student/PhD). Key components of any dataset and benchmark paper should be in the main copy. Please See How IndustryEQA has written the key components. In Line 214, the paper mentions “Each of the 21 templates…”… I could not find where it is mentioned that a total of 21 templates were developed. Lines 211-213 point to Appendix Sections D, E, and F. The key point must be brought back to the main text; this is the art of writing an effective, self-contained paper. It doesn't tell me how I would implement the methodology if I read, say, Section 3.1, Section 3.2, and Section 3.3.
-
Long paragraphs on page 4 and page 5 need attention. Flow is an issue. For every section, give an outline of what should be discussed or expected at that point in time. For example, look at sections 4.1, 4.2, and 4.3; the headers don't show how these three are interconnected. Maybe there is a reference in Figure 1.
-
Line 263 mentions that a PhD-level expert created a dataset. This needs a study of human labeling error or bias. Section 4 doesn't justify its availability; is the use of CFG an innovation, or is it already used? Including one example of each task type in the main paper could have been helpful. This also raises another question: did the authors conduct a similarity search between their question and, say, questions from ITFormer or AssetOpsBench, and note any differences?
-
Section 5.1: again, many references to Appendinx. It was unclear how TSFM models answered Level 1 questions. My understanding is that the question goes to the LLM agent, and it may decide to use the TSFM model for any analysis. Is my understanding clear? In a typical dataset paper, after preparing the DB, you show some statistics, like the number of options. Especially in MCQA, how do you ensure your distractors are sampled appropriately? I found the DiagnosticIQ paper that describes the pipeline well and provides a detailed dataset distribution of options, etc. I'm currently reading the research paper on page 6, and it’s not very clear.
-
Section 5,2, First result. I haven't seen anything: no leaderboard, no results, and the conclusion is highlighted in bold. Line 295-296. Please discuss the parameters that are used for the LLM. Then walk us through the overall result, and then give some insights. For 5.2, I need an example to understand the different failure modes.
-
The fonts in Figure 2 (b) are not easily readable. The purpose of Factory vs. Robot needs to be clearly articulated. IndustryEQA uses Factory as an environment to create Q&A and solve problems; however, here the base-row data comes from Robot. The model cites many time series models; however, the main paper includes no in-depth analysis.
-
The paper abstract includes a link to a Hugging Face repo that, in my opinion, lists the IDs or names of people associated with HF repositories. I will ask the area chair to investigate whether this violated the double-blind policy. Kaggle provides a clean anonymous way of sharing the data. I highly recommend it for future submissions.
Questions for authors
I have read the paper and have no question that can change my assessment.
6 comments
Ack and More Feedback (on revised manuscript)
I acknowledge that I read the rebuttal response. I have a few pieces of feedback on the (revised version) of the paper
- Kindly note that in the revised manuscript, we cannot say which parts are modified and which parts are original. Ideally, many authors use different colors to show the changes.
- I always suggest not including a laundry list of the paper (lines 069-070). As said earlier, writing the paper requires a careful presentation of the related work instead of just listing it.
- Font size in the table (in the main paper ): I expect the font size to be as close to the main text as possible. Readability improves. (Tables 1, 2, 3). Space is not an issue, 10th page is 50% empty.
- Section 3 and Section 4 seem reformatted, but their quality/representation may need more work.
- \cite vs \citep - pls prefer to use \citep,
- Thanks for the revised manuscript; it has improved a lot based on the review.
Second revision: changes marked in colour, citation lists in the introduction improved, larger tables, shorter paragraphs
Thank you for reading the revision and for the further feedback. We adopted all of it.
Marked changes. The supplementary material now contains the paper with every change from the original submission marked: added text in blue, removed text struck through in red.
Citation lists in the introduction (lines 069–070). The list of forecasting architectures is now a sentence that says what each line of work adds: the Transformer adapted to long horizons and to patches of the series, pretrained zero-shot forecasters such as Chronos and TimesFM, and self-supervised event prediction. The list of time-series Q&A benchmarks in the next paragraph is rewritten the same way: TimeSeriesExam and TSAQA ask about univariate series; ChatTS, Time-MQA, MTBench and QuAnTS use multivariate series that are not recorded from a machine; and the closest, EngineMT-QA, uses simulated turbofan telemetry rather than a robot under closed-loop control.
Table fonts. Tables 1–3 font size is increased (9pt, up from 5 to 7pt). Table 2 is no longer scaled down, and its intervals sit under each value. The appendix tables are now set at 9 pt as well.
Sections 3 and 4. Following your original review, the headings now show how the parts connect: 3.2 "Signals and the setpoint–context–effort schema", 4.1 "Templates: from episode labels to questions and answers", 4.2 "Fault and counterfactual recordings in FactoryWave" and 4.3 "Level 4 references from manufacturer documentation". Section 3 now opens with an outline, as Section 4 does, and the two longest paragraphs of Section 4.1, of 16 and 25 lines, are each split in two.
Citations. All citations now use \citep.
Now self-contained, positioned against 8 benchmarks, with no data or question overlap
Thank you for the detailed review. Following it, Section 2 now positions FactoryBench against eight industrial and time-series QA benchmarks, the main text carries the grammar, one example per level and a leaderboard, and we show that FactoryBench shares neither data nor questions with EngineMT-QA or AssetOpsBench.
Related industrial benchmarks. Section 2 now discusses AssetOpsBench, IndustryAssetEQA, SenTSR-Bench, FailureSensorIQ, DiagnosticIQ, IndustryBench, IndustryEQA and EngineMT-QA (ITFormer). What sets FactoryBench apart is its method, not only its robots: Level 1–3 answers are computed from recorded multichannel telemetry with physically injected faults, counterfactual answers come from matched physical reruns, and the four levels run from reading the machine state to choosing a repair. We also corrected lines 122–124: AssetOpsBench and IndustryAssetEQA do evaluate LLM reasoning over industrial telemetry, and the sentence now states the actual difference.
Question generation and Self-Instruct. Our generator uses no LLM. Level 1–3 answers are computed from recorded signals and fault labels, and Level 4 references come from manufacturer documentation and expert protocols. We claim no novelty for the template grammar itself; the contribution is checkable labels from physical recordings.
Grammar, the 21 templates, appendix content. Section 4.1 now defines the grammar (template, typed slots, slot sources) with a worked derivation, states the 21 templates and summarises Appendices D–F.
Examples, statistics, regeneration, industry tasks. Table 1 shows one example per level, Section 4.1 carries the technical detail of generation, and Section 3.2 now points to a new statistics table covering episodes, robots, sampling rates, tasks, faults and items per level and format (Table 8, App. C). Section 4.1 names the regeneration entry points, and Section 3.1 maps the four levels to condition monitoring, impact assessment, what-if analysis and troubleshooting.
Figures. Figure 1 is redrawn at print size (labels 7 to 8.6 pt, up from 3.3 to 4.5 pt) and reads in greyscale; Figure 2b is redrawn for all six models (Figure 12).
Citations. Six references now cite the published version: PatchTST, Time-MQA, ITFormer, QuAnTS, Peyrard & West and TimeSeriesExam. TSAQA has no published version yet.
Human labelling. We added a blind check: two external robotics engineers score 0.75 and 0.70 raw on 30 rebuilt Level 3 items against 0.33 for the best model (Table 37). Level 1–3 labels are computed, not annotated; a released 102-item expert solution set is audited ground truth, and an independent solver blindly re-solving 20 Level 1–2 items reproduced 19 stored answers (App. G.2).
Overlap with ITFormer and AssetOpsBench. FactoryBench shares neither data nor questions with these benchmarks. EngineMT-QA is built on simulated turbofan engines and AssetOpsBench on chiller telemetry, whereas FactoryBench uses recorded industrial robot arms with injected faults. A similarity search confirms this for the questions themselves: across all 949 Lite question stems, the highest similarity to any of 14,747 EngineMT-QA and 440 AssetOpsBench questions is 0.14 in wording (TF-IDF) and 0.52 in meaning (MiniLM), against a median of 1.00 between EngineMT-QA's own test and train questions (App. Q.1, Table 35).
Section 5: leaderboard, decoding, forecasters, options. Section 5.2 now opens with a leaderboard (Table 2) that includes a trivial baseline per template, and Section 5.1 states the decoding (temperature 0, one message, 2,000-token cap). Section 5.1 now adds the time-series models TimesFM-2.5 and Chronos-Bolt as standalone forecasters, not LLM tools: they score level with Claude on 166 L1.7 items (+63.1 and +60.6 vs +61.4), and TimesFM-2.5 beats it on Lite L1.7, L2.4 and L2.5 (+37.9/+15.2/+24.9 vs +34.9/+3.0/+18.3; Table 22). Section 4.1 gives the number of options and how distractors are sampled, unlike DiagnosticIQ's embedding-based sampling, and Section 5.2 now walks from the overall result to the per-level findings with a failure example.
Factory and robot. Section 1 now explains the name: the data are telemetry from industrial robots working in factory cells, rather than a factory scene as in IndustryEQA.
Hugging Face link. We understand the concern, and to our knowledge the link does not identify us. The names on the page appear to be Hugging Face's list of users who liked the dataset; none of them is an author, and the dataset card and Croissant metadata name no author or affiliation. Kaggle is a good suggestion that avoids this issue, and we will use it for future anonymous releases.
Thank you: these suggestions made the paper self-contained and place it among the industrial benchmarks you named.