- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors present factorynet, a multi-embodiment industrial time-series corpus spanning 56,591 end-to-end task executions (simulated & real), 113M logged timesteps, 7 embodiments, 4 tasks, 27 annotated anomaly types with healthy baselines and counterfactual pairs. An embodiment is a distinct machine platform (e.g. a different robot arm). The benefit of this collection is, that it not only contains the resulting physical state signals of machines, but rather the full control loop, including targets and actuation effort, properly labeled and structured. This enables one to structurally separate cause and effect and to work with the commanded-versus-realized dynamics measured during recording.
In the presented collection, every signal is mapped to a machine-agnostic control-theoretic schema: setpoint (commanded intent, e.g., target joint positions), effort (actuation, e.g., motor current or torque), feedback (measured outcome, e.g., actual position), context (environment, e.g., payload mass). Additionally, they present two studies, (1) they investigate if the schema supports anomaly detection, and (2), they investigate transfer of a monitoring model between datasets (sim to real and robot to robot).
(1) instead of reconstruction-based anomaly detection, they fit an MLP to map setpoints to efforts and compute the per-episode mae between the estimate and real effort as their anomaly score. Their approach lands mid-pack. Switching to a mapping from setpoint to feedback ends up being the strongest approach in their evaluation. They argue that their schema can serve as a reduced search space to select input and target channels for anomaly detection.
(2) They test two tasks: phase identification (which stage of the pick-and-place cycle is running) and binary fault detection. The presented results are interesting and highlight the benefits of having real and simulation data in the collection.
Overall the paper is in a good shape and is a timely and important contribution for deep learning research in the manufacturing sector. The work took notable standardization efforts and presents a senseful schema for future dataset releases that contain closed-loop episodes of manufacturing machines. The huggingface page is well documented. Having that many fault episodes in the collection is indeed interesting for anomaly detection and robustness research, but the ratio seems a bit skewed towards faulty data (72/28). To improve the presentation I recommend to add a brief list of contributions to the end of the introduction and to add a visualization of one episode of the dataset to the main paper. The anomaly detection experiments could be improved with more recent baselines, more details and metrics. Since the main contribution is the presented collection and standardization effort, I recommend acceptance.
Critical strengths
- The curated dataset and structuring is exactly the standardization effort the field needs for the work on foundation models and more general assessment of algorithmic improvements across use cases and embodiments.
Critical weaknesses
None
Questions for authors
- What exactly was the bootstrapping setup that you mentioned in your paper?
- Could you provide additional details about the anomaly detection evaluation setup? (e.g. scoring, score aggregation, additional metrics)
- You hypothesize that existing TSFMs can't disentangle static payload biases from dynamic behavior, have you performed experiments on that?
We thank the reviewer for a careful review. The central point is fair: the paper evaluated several tasks without naming them, which made the problem look as if it shifts. The revision names them at the start of Section 4.
What is the relationship between fault detection and anomaly detection? Is the paper focusing on fault detection or anomaly detection?
A single score over all signals hides this structure; residuals between roles expose it.
The remaining points follow in the next comments.