- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper is about industrial fault diagnosis when the sources are heterogeneous: acoustic, vibration, current and voltage records with different sampling rates, different fault vocabularies, and different sets of annotated fields. Each query comes with machine context and a candidate list, and the system has to pick a condition and fill whatever record fields the source supports. The proposed setup takes a frozen FISHER embedding of the query, q, retrieves a healthy prototype r from a train-only bank by cosine similarity (up to 16 groups, one view per group), and feeds [q, r, q-r, |q-r|] to one of two heads. RELATE-MLP is a 6.30M-parameter MLP over the full 24,576-d representation with setting-local candidate masks. RELATE-Gen adds a MiniROCKET branch, fuses the two through a gated residual block, squeezes the result into 4 continuous tokens, and uses them to condition a frozen 4-bit Qwen2.5-7B-Instruct with LoRA that generates the record as JSON.
The benchmark, MOSAIC, is 11 RMIS settings plus one MIMII-DG setting with frozen group-level splits, 15,477/1,218/3,598 groups. On test, RELATE-MLP gets 80.60/78.36 setting-macro accuracy/F1 and RELATE-Gen 78.65/77.01, with 68.86 Joint EM after per-field voting and 79.95 Field EM. A readout on the same frozen tokens gets 69.07 Joint EM at 45.95 views/s against 0.20 for generation. Three-run validation controls give +5.07 for the healthy reference and +4.56 for the temporal branch with fusion; single-run controls cover token count, input blocks and prefix size. There is also a CWRU adaptation compared against a locally adapted BearLLM.
Critical strengths
-
The classifier result holds up under both aggregations the paper reports, and the harness behind it is unusually careful. Every method, including the three proprietary zero-shot APIs, is scored by the same group-level scorer with the same denominators and metric weights. The voting rules are written down rather than left implicit: missing values enter the per-field vote and only win if they beat every non-missing value, ties among non-missing values break by canonical JSON lexical order, a field is emitted only when its winning value is non-missing, and paths outside a group's supported set are dropped from the record metric but counted in a separate audit, which reports that "none of the 11,171 emitted group-level field paths fell outside their corresponding setting-level schemas". Gold is joined after generation, and the prompt carries no dataset IDs, signal UIDs, group IDs or field mask. I recomputed the tables instead of trusting them: the twelve per-setting rows of Table 1 reproduce the setting-macro row to within 0.005 in every column, both panels of Table 11 and all four columns of Table 10 reproduce theirs, the twelve rows of Table 8 sum exactly to 15,477/33,476, 1,218/2,079 and 3,598/4,430, and the two within-condition fields of Table 23 average to exactly the Severity EM column of Table 2. What is solid is that RELATE-MLP's lead does not depend on the weighting questions I raise below: it is +5.36 on setting-macro and +2.00 on group-micro, and it is ahead on 10 of 12 settings. A 6.30M-parameter model whose whole route bundle trains in about ten minutes on one consumer GPU and beats every baseline on both aggregations is the strongest thing in the paper. To be clear about where the value is: most of the machinery is borrowed, including the query-reference-residual construction that Section 2 attributes to BearLLM, so the contribution worth having is the protocol, the harness, and the finding that a frozen representation plus a small MLP is enough to lead the table. That is a real result, but it is not a new architecture.
-
The paper is candid about the results that go against it, and the controls are matched properly. FISHER 1-NN is reported as beating the generator on group-micro (89.83 vs 88.85), the classifier-minus-generator interval is reported as spanning zero, going from 4 to 8 tokens is reported as hurting, the expanded prefix is reported as worse than the compact one, the auxiliary-loss control is reported below the autoregressive objective, and the input control is reported as favouring [q, r] over the full four blocks. The controls themselves are better matched than I am used to seeing here: the three input arms share one saved initialisation and differ only in masking, the AR and Direct interfaces share features, sampler, schedule and endpoint, and Qwen and its readout receive literally identical token vectors exported from the same checkpoint. I raise this as credit, but it also shapes how I read the paper. The contradictions I complain about below are between the abstract and your own tables, not between the paper and reality, which is why my concerns are about framing and coverage rather than about the work itself.
Critical weaknesses
- The arithmetic behind the headline runs one way for the classifier and the other way for the generator, and the abstract reports only the setting-macro side of both. Setting-macro gives each setting one twelfth, but the settings are not remotely the same size: Table 8 has 16 test groups for umged_cur, 28 for umged_vol, 36 each for pu_cur and pu_vib, and 1,000 for mafaulda_vib, so the biggest is 62.5x the smallest. Since every per-setting accuracy is a count over n groups, every margin in Table 1 is an exact multiple of 100/n, so the margins can be converted back into groups. I did that. RELATE-MLP comes out +72 groups against FISHER 1-NN, which is exactly what its group-micro row implies (+2.00% of 3,598 = 72). RELATE-Gen comes out -35 groups (-0.98% of 3,598 = 35), so the same Table 1 that gives it +3.41 points of setting-macro gives it 35 fewer groups correct, and it is ahead on only 5 of 12 settings. Four small settings carry 83% of the classifier's macro lead and 118% of the generator's; for the classifier the group-level count agrees with the macro, so the concentration does not change its sign, and for the generator it does. Your own intervals say the same thing. RELATE-Gen minus 1-NN is "+3.41 [0.02,6.94]", whose lower end is two hundredths of a point, and the two settings carrying that margin have intervals spanning more than 28 points each (PU current +30.56 [13.89,47.22], UMGED voltage +21.43 [7.14,35.80]). The consequence is narrow but it matters for contribution (3): the claim that the classifier leads the generator rests on "[-0.56,4.60]", which crosses zero. On the record side, Field EM is the mean of seven paths and one of them is handed to the model: Appendix G says "Machine type is provided in the prompt", and on the Table 24 validation rows machine type scores 99.94 and 100.00 against a seven-path mean of 78.17, so dropping that one path would leave 74.54, with three of the remaining paths described by the authors themselves as "largely determined by the primary taxonomy". Severity EM (52.48 against 51.78) has no interval row in Table 13 at all, and Table 23 has the frozen-token readout beating the generator on eccentricity, 32.89 against 27.63. That last comparison is like-for-like, since the taxonomy mapping only covers non-severity fields.
None of this is a numerical error, and I want to say that plainly because my recommendation leans on the tables being trustworthy. What I would ask for here is a re-scoring. One comparison in Table 2 is not like-for-like, and there the asymmetry runs against the generator rather than in its favour: the caption says both readouts use the same train-only taxonomy rules, and those rules fill in the non-severity fields that the primary condition determines uniquely, whereas the generator predicts the complete record itself. Its 68.86 Joint EM is therefore not directly comparable to the readout's 69.07, and until the generator is scored under the same mapping the paper cannot say what the generative route is worth. That is my first question below.
- Each component claim rests on a single three-run comparison, and both numbers move when the recipe moves. Section 5 says "Validation selects RELATE configurations". The frozen-test tables contain no ablation rows at all: no query-only, no reference-free, no other objective. Every component comparison sits on the same 1,218 validation groups that chose the components, in Tables 3, 4, 19, 20, 21 and 22, and every one of them except the two headline ablations is single-seed. The headline ablations themselves are three-run, but the auxiliary-loss versions of the same two are single-seed, and both come out far lower: the reference at +0.59 against +5.07 (71.20 to 71.79 in Table 22), and the temporal branch with fusion at +1.37 against +4.56 (70.42 to 71.79 for FISHER-small in Table 21). With three runs on one side and one on the other I cannot tell whether that gap belongs to the objective or to the seed, and the abstract carries neither the numbers nor the caveat. There is a second issue in how the reference ablation was built. The control "keeps the temporal branch and input geometry, replacing the FISHER blocks with [q, q,0,0]", which zeroes the difference blocks as well, so the +5.07 credited to the reference also contains whatever the difference blocks contribute; Table 20 measures those separately at +2.43. The same tension runs through the method. Eq. (1) and Figure 1(b) define the input as [q, r, q-r, |q-r|], but in the matched control [q, r] gets 78.38/76.79 against 77.31/75.08 for the full four blocks, so the blocks the framework is named after cost 1.07 and 1.71 points, and the main model keeps all four anyway while reporting 78.38, the same number the two-block arm reaches. Some choices are also being made inside your own error bars: 4 tokens is selected over 8 (75.75 against 75.36, a gap of 0.39) against standard deviations of 1.20 to 1.90 that the paper reports across its three-run configurations, on a sweep the paper calls "single-seed". None of this overturns the classifier result, which is a frozen-representation result and does not depend on the autoregressive objective at all. It does mean that contribution (1), the claim that the reference and the fusion are what makes the method work, is supported only on the split that selected the configuration, and by one comparison per component.
That is why I would reject the paper as written rather than ask for a prose revision. What the paper claims is that the health-relative components are what makes the method work, and that the generative route holds its own against the non-language alternatives. On the evidence submitted neither is established: the component gains come from the split that selected the configuration, the auxiliary-loss versions of the same two ablations come out several times lower (single-seed, Tables 21 and 22), the difference blocks that Eq. (1) is built around cost accuracy in the paper's own matched control, the classifier-minus-generator interval crosses zero, and the generator's macro lead rests on 3.2% of the groups. What does hold is the protocol, the harness, and the fact that a frozen representation plus a small MLP leads every aggregate metric. The protocol is claimed as contribution (2) but not delivered, while the harness and the classifier result are delivered but never claimed. So the three things I would need are a claim and a delivery that match the evidence: re-frame contributions (1) and (3) around the classifier result and the harness, scoring the generator under the same taxonomy rules as the readouts so that anything still claimed for it can be checked like for like; release the split manifests and the evaluation code, without which contribution (2) cannot be reproduced; and report per-setting n with per-setting intervals beside a group-weighted column, carrying the auxiliary-loss numbers next to the autoregressive ones wherever the abstract's two gains appear. I would reconsider on those three, and none of them needs a new model: two are writing and release work, and the third is a re-score and arithmetic on data you already have.
Questions for authors
Q1. Table 2's caption says both readouts use the same train-only taxonomy rules, but the generator predicts the complete record itself, so its 68.86 Joint EM is computed under a harder rule than the readout's 69.07. Can you re-score the generator under the same taxonomy mapping so that the three rows are comparable? And can you break its remaining errors into malformed JSON, out-of-candidate primary labels, and well-formed records with wrong field values? If format dominates, grammar-constrained decoding would remove most of the gap and the 4-token interface would have to be argued on other grounds.
Q2. Can you report the healthy-reference ablation and the [q, r] versus [q, r, q-r, |q-r|] comparison on the frozen test split, rather than only on the split that selected the configuration? If that is not available, what should a reader conclude about contribution (1) from validation-only evidence?
Q3. Table 22 has the auxiliary-loss version of the reference ablation (+0.59) and Table 21 the auxiliary-loss version of the fusion ablation (+1.37), against +5.07 and +4.56. The autoregressive numbers are three-run and the auxiliary ones single-seed, so I cannot separate the objective from the seed. Can you run the auxiliary variant with three runs, or state that both gains are conditional on the objective? Related: since the no-reference control zeroes the difference blocks as well as swapping the FISHER blocks for [q, q,0,0], could you add a control that removes the reference while keeping the four-block geometry?
Q4. FISHER 1-NN beats the generator on group-micro, and the generator's macro lead sits in four settings holding 116 groups. Could you give per-setting test n and per-setting intervals, plus setting-macro under group weighting? Which of the two aggregate conclusions should a reader carry to an unseen setting, and does the protocol in contribution (2) commit to one of them?
Q5. Will the MOSAIC split manifests, benchmark-group identifiers and evaluation code be released? Section 3 points to Appendix A for "the split construction and source identifiers", but Appendix A defines the source-family keys rather than listing identifiers, and Appendix A.1 describes a class-balanced subset drawn by quota from that partition rather than the full native test set. For a benchmark contribution I would need the manifests themselves.
Q6. Why does the main RELATE-MLP keep the four input blocks when the matched control gives them -1.07/-1.71, and is its 78.38 distinguishable from the 78.38 of the [q, r] arm, given that the two use different recipes (group-level 80 epochs against view-level 100,424 steps)?
Q7. RELATE-CWRU trains with an auxiliary candidate-masked cross-entropy head over the ten classes, removed at inference, and the description of the locally adapted BearLLM baseline does not mention a corresponding supervised head, on 222 test segments with different base-model sizes. Can you either align the two recipes or restrict the conclusion to what the aligned part supports?
4 comments
Response to T9xv (2/4): Q2–Q3 — three-seed controls
Continuing our response to Reviewer T9xv.
Q2 Input controls on the frozen test split
We evaluated the MLP and Gen three-arm input comparisons on the frozen test split across three training seeds. The results use setting-macro accuracy on the same 3,598 groups and 12 settings.
| MLP input | Test accuracy mean±SD |
|---|---|
| Query only [q,0,0,0] | 79.15±2.08% |
| Query/reference [q,r,0,0] | 78.89±1.74% |
| Full [q,r,q-r,abs(q-r)] | 79.45±0.90% |
Within a seed, these MLP arms share the saved initialization, architecture, 80-epoch recipe, sampler and 24,576-dimensional input width. Full minus query-only averages +0.30±2.36 points, and full minus pair averages +0.56±1.28. The explicit differences have a small mean effect for this classifier, with variation across seeds.
| Gen input | Test accuracy mean±SD |
|---|---|
| Self-reference [q,q,0,0] | 74.96±1.03% |
| Query/reference [q,r,0,0] | 75.92±0.82% |
| Full [q,r,q-r,abs(q-r)] | 76.76±1.72% |
Full input improves accuracy in each seed, with a paired gain of +1.80±1.47 points over self-reference. Retaining the healthy reference and masking only the differences gives 75.92±0.82%, versus 76.76±1.72% for full input. The paired full-minus-pair accuracy difference is +0.85±2.28 points. The individual differences are as follows.
| Training seed | Full-input test accuracy | Pair-only [q,r,0,0] test accuracy | Full minus pair-only, pp |
|---|---|---|---|
| 111 | 78.65% | 75.17% | +3.48 |
| 222 | 75.28% | 75.78% | −0.50 |
| 333 | 76.35% | 76.80% | −0.45 |
Pair-only is 0.70 points below self-reference at seed 111. Across three seeds, full and pair-only F1 is 75.07±1.75% and 74.96±0.88%, with a paired difference of +0.11±2.44 points.
Gen uses the fixed 100,424-microstep endpoint, greedy JSON generation and the same group scorer. These frozen-test comparisons supplement the validation evidence and distinguish the complete reference construction from the difference blocks with the reference retained.
The complete frozen-test controls are in revised Appendix F, Tables 28–29.
Q3 Objective dependence and the four-block control
The reference and fusion effects depend on the training objective. Both auxiliary comparisons now have three seeds at the same fixed 100,424-microstep endpoint. All entries below are validation setting-macro accuracy differences on 1,218 groups and 12 settings.
| Contrast | Autoregressive objective, three seeds | Auxiliary objective, three seeds |
|---|---|---|
| Full versus self-reference | +5.07 pp | +0.95±1.32 pp |
| Temporal branch with learned fusion versus FISHER-only | +4.56 pp | −0.58±2.10 pp |
For the reference comparison, AR full/self-reference reaches 74.73±1.20% / 69.66±1.79%, with a positive gain in each seed. Auxiliary full/self-reference reaches 71.39±0.43% / 70.44±1.66%. Its paired differences are +0.5903, +2.4048 and −0.1548 points, averaging +0.95±1.32.
For auxiliary-objective fusion at the same 100,424-microstep endpoint, full and FISHER-only accuracy is 71.39±0.43% and 71.97±2.01%, respectively. Their paired differences are +1.37, −0.30 and −2.80 points at seeds 111/222/333, averaging −0.58±2.10 points. These ± values are sample standard deviations across training seeds. The fusion comparison therefore gives different results under the two training objectives.
The self-reference control retains the four-block formula by setting r*=q. Then [q,r*,q−r*,abs(q−r*)] is [q,q,0,0]. Input width, temporal branch and projection are held fixed. The resulting gain measures the whole healthy-reference construction, including the deviations derived from the reference. To distinguish the explicit differences, [q,r,0,0] keeps the healthy prototype and masks only those two blocks. Full minus this pair-only arm measures the difference blocks with the reference retained. We report these contrasts separately because they answer different questions about the construction.
The three-seed auxiliary-objective results are in revised Appendix F, Tables 30–31.
Response to T9xv (3/4): Q4–Q5 — weighting and reproducibility
Continuing our response to Reviewer T9xv.
Q4 Weighting, sample sizes and uncertainty
The protocol uses group-level setting-macro accuracy as the primary metric, giving each diagnostic setting equal weight. Group-micro accuracy gives each test group equal weight. The fixed submitted models give the following results on the same 3,598 groups.
| Fixed model | Setting-macro accuracy | Group-micro accuracy |
|---|---|---|
| FISHER 1-NN | 75.24% | 89.83% |
| Main RELATE-MLP | 80.60% | 91.83% |
| Seed-111 RELATE-Gen | 78.65% | 88.85% |
The MLP gains 72 correct groups over 1-NN and leads on 10 of 12 settings. Its paired gains are +5.36 macro points [2.51, 8.43] and +2.00 micro points [1.11, 2.86]. The MLP-minus-Gen macro difference is +1.94 [−0.56, 4.60]. These results support the full-evidence classification route under both benchmark weightings.
The fixed Gen model leads 1-NN on 5 of 12 settings and has 35 fewer correct groups overall. Its macro difference is +3.41 [0.02, 6.94], while its micro difference is −0.98 points. The macro gain is concentrated in small settings. Both weightings are needed to interpret that result. Per-setting group counts and intervals are given here.
| Setting | n | MLP accuracy [95% CI] | Gen accuracy [95% CI] | 1-NN accuracy [95% CI] |
|---|---|---|---|---|
| idmt_air | 400 | 89.50 [86.50, 92.25] | 85.75 [82.25, 89.00] | 89.50 [86.50, 92.26] |
| mafaulda_sound | 290 | 85.52 [81.38, 89.32] | 82.07 [77.59, 86.21] | 80.00 [75.17, 84.14] |
| mafaulda_vib | 1000 | 97.60 [96.60, 98.40] | 93.20 [91.60, 94.70] | 97.50 [96.50, 98.40] |
| pu_cur | 36 | 61.11 [44.44, 77.78] | 75.00 [61.11, 88.89] | 44.44 [27.78, 61.11] |
| pu_vib | 36 | 83.33 [69.44, 94.44] | 83.33 [69.44, 94.44] | 80.56 [66.67, 91.67] |
| sdust_bearing | 400 | 97.50 [96.00, 98.75] | 97.00 [95.25, 98.50] | 97.25 [95.75, 98.75] |
| sdust_gear | 700 | 100.00 [100.00, 100.00] | 99.29 [98.71, 99.86] | 99.43 [98.86, 99.86] |
| umged_cur | 16 | 62.50 [37.50, 81.25] | 43.75 [18.75, 68.75] | 50.00 [25.00, 75.00] |
| umged_vib | 32 | 59.38 [43.67, 75.00] | 68.75 [53.12, 84.38] | 68.75 [53.12, 84.38] |
| umged_vol | 28 | 60.71 [42.86, 78.57] | 60.71 [42.86, 78.57] | 39.29 [21.43, 57.14] |
| wt_plane_gearbox | 60 | 90.00 [81.67, 96.67] | 80.00 [71.63, 90.00] | 81.67 [70.00, 90.04] |
| mimii_dg | 600 | 80.00 [76.67, 83.17] | 75.00 [71.50, 78.50] | 74.50 [70.83, 78.00] |
The per-setting intervals use the existing 1,000-replicate percentile bootstrap, with benchmark groups sampled within settings and the seed-111 models fixed. The paired aggregate intervals use 10,000 shared within-setting group resamples. Training-seed variation is reported separately as SD. A perfect observed result, such as the MLP on SDUST gear, produces a degenerate empirical-bootstrap interval.
For the matched three-seed MLP controls, query-only, pair-only and full input have group-micro accuracy of 91.15±0.56%, 90.73±0.26% and 91.74±0.16%. Full adds 44, 32 and 33 correct groups over pair-only, averaging +1.01±0.19 micro points. Their setting-macro difference remains +0.56±1.28. Setting-macro is the protocol's primary summary for equal weighting of the twelve known settings, while group-micro describes their observed test mixture. Evaluating transfer to an unseen setting requires a held-out-setting experiment.
Revised Appendix B, Table 11, lists the per-setting sample counts and intervals.
Q5 Release of the actual benchmark materials
Yes. We will release the exact train/validation/test group and view manifests, benchmark-group and view identifiers, fixed candidate order, label/schema definitions and executable scoring code in anonymous supplementary materials. The manifests specify 15,477/1,218/3,598 groups and 33,476/2,079/4,430 views. They identify the actual class-balanced quota subset used here, including its relationship to the upstream RMIS partition.
The materials also contain Gen predictions for all 4,430 seed-111 test views, with gold annotations stored separately. The README provides commands for primary scoring, raw structured scoring and the common train-only taxonomy rule. Those commands reproduce Joint EM of 68.8642% and raw/mapped Field EM of 79.9503/79.9368%. Source attribution, upstream conditions and a dated environment snapshot accompany the scorers.
Response to T9xv (4/4): Q6–Q7, fields and token count
Continuing our response to Reviewer T9xv.
Q6 Why expose the differences, and which MLP recipe is used?
We retain the four blocks to give both predictors the query state q, healthy-reference state r, signed deviation q−r and deviation magnitude abs(q−r) as explicit inputs. The MLP uses the complete representation, while Gen compresses it into four tokens after temporal fusion. The difference blocks are deterministic functions of the pair, and their measured contribution is evaluated separately for each predictor below.
The matched three-seed MLP validation accuracy is 76.08±1.89% for full input and 76.16±1.97% for pair-only. Its test full-minus-pair difference is +0.56±1.28 points. Pair-only is competitive for this classifier. The Gen results in Q2 distinguish the complete reference construction from the explicit difference blocks with the reference retained.
The two 78.38% validation values are from separate training runs under the same nominal group-level, 80-epoch primary-MLP recipe. The matched input arms share a saved initialization and sampler. Input effects are estimated within those arms. The 100,424 view-microstep recipe in the neighboring appendix paragraph belongs to the separate structured-MLP experiment.
| Experiment | Training recipe | Validation accuracy | Test accuracy |
|---|---|---|---|
| Submitted main full MLP | Group-level, 80 epochs | 78.38% | 80.60% |
| Matched query/reference control | Same saved initialization and 80-epoch control recipe | 78.38% | 77.64% |
| Matched full-input control | Same saved initialization and 80-epoch control recipe | 77.31% | 79.57% |
The main classifier's 80.60% test result is its fixed-run benchmark result. The matched full-input result of 79.57% is used with the other matched arms to estimate input effects.
Revised Appendix F, Tables 27–28, report the fixed-seed and three-seed input controls separately.
Q7 Scope of the CWRU comparison
We restrict the conclusion to the two complete local adaptations evaluated on the same 222 one-second CWRU test segments and ten-class candidate labels. RELATE-CWRU uses a frozen FISHER-relative signal branch, Qwen2.5-7B and a training-only auxiliary candidate-masked primary-condition head. Adapted BearLLM uses its released signal components and Qwen2.5-1.5B. The auxiliary head is removed for RELATE-CWRU inference.
| Adapted system | Segment accuracy | Segment macro F1 |
|---|---|---|
| RELATE-CWRU | 83.33% | 80.95% |
| BearLLM | 43.24% | 31.49% |
The observed advantage applies to these systems under their respective adaptation recipes. Encoder, model size and supervision are part of that comparison.
The two adaptations are documented in revised Appendix C, Table 15.
Supplied fields, severity uncertainty and token count
Machine type is given in the prompt. For seed-111 Gen it is correct on all 600 supporting test groups. Removing that field from the raw seven-field average changes Field EM from 79.95% to 76.61% over the other six fields. Health state, location and mechanism largely express the condition taxonomy. We therefore report those record fields separately from damage size and eccentricity, which can vary within a condition, and use the common taxonomy rule in Q1 for the three methods.
Across three seeds, Gen-minus-AR severity results use all 300 damage-size groups and 76 eccentricity groups in each seed. Severity EM weights the two fields equally.
| Field | Gen mean±SD | AR readout mean±SD | Paired Gen−AR difference, pp |
|---|---|---|---|
| Damage size | 74.44±2.83% | 72.11±1.50% | +2.33±3.79 |
| Eccentricity | 24.12±4.02% | 32.02±2.74% | −7.89±5.74 |
| Equal-field Severity EM | 49.28±2.85% | 52.06±1.75% | −2.78±3.84 |
The damage-size differences are +6.67, +0.67 and −0.33 points across seeds. Eccentricity favors the readout in all three. For the fixed seed-111 models, Gen obtains 77.33% damage-size and 27.63% eccentricity EM, versus 56.00% and 23.68% for the learned MLP severity heads. The paired 95% intervals below use 10,000 within-setting benchmark-group resamples and identical draws for the compared models.
| Fixed seed-111 contrast | Damage-size EM difference, pp [95% CI] | Eccentricity EM difference, pp [95% CI] | Severity EM difference, pp [95% CI] |
|---|---|---|---|
| Gen minus AR readout | +6.67 [2.31, 10.97] | −5.26 [−13.16, 2.63] | +0.70 [−3.95, 5.26] |
| Gen minus MLP severity heads | +21.33 [15.36, 27.33] | +3.95 [−7.89, 15.79] | +12.64 [5.85, 19.47] |
In the matched single-seed token sweep, four tokens reach 75.75% validation accuracy and eight reach 75.36%, a 0.39-point difference. We choose four for half the signal-prefix length with slightly higher observed validation accuracy in that sweep. Separately, the final four-token configuration has a three-seed validation accuracy SD of 1.20 points. That SD describes repeated training at four tokens.
Response to T9xv (1/4): Q1 — comparable record scoring
Thank you for your careful review and constructive feedback.
Substantive revisions in the manuscript are marked in blue.
Q1 Same-taxonomy scoring and remaining errors
Yes. We applied the same training-only field-filling rule to all three predictors. For the originally reported seed-111 Gen model, Joint EM remains 68.86% under this common rule. Gen and the AR readout use the same signal tokens learned through RELATE's structured-record generation training. Direct learns its signal tokens through direct condition and field supervision. All three cover 4,430 views, 3,598 benchmark groups and 12 settings in each training seed. The three-seed results are as follows. Here ± denotes sample standard deviation across seeds 111/222/333.
For the original seed-111 comparison, raw/mapped Joint EM is 68.86/68.86% for Gen, 68.47/69.07% for AR and 65.08/66.70% for Direct. Gen raw/mapped Field EM is 79.9503/79.9368%.
A paired comparison of all 3,598 groups finds 29 more fully correct Gen records than AR records at seed 111, 13 more at seed 222 and one fewer at seed 333. These are equally weighted group counts. The table's Joint EM gives each setting equal weight. The three UMGED settings contribute −1.25 points to the three-seed mean Gen-minus-AR Joint EM difference, while the other nine settings together contribute +0.48 points. Both summaries use the same common-mapped predictions.
The seed-111 error decomposition contains zero malformed JSON records and zero out-of-candidate primary labels among all 4,430 generated records. After aggregation, 3,110 groups are fully correct. Among the other 488 groups, 401 have a wrong primary condition, including two that also lack a severity field. The other 87 have the correct condition and a wrong severity value. These are group counts, whereas Joint EM averages the twelve setting-level proportions equally. Condition and severity estimation account for these errors even when the output is syntactically valid, so grammar constraints alone would leave their main cause in place.
With the same readout architecture, the AR-trained representation gives higher test Joint EM than Direct in all three seeds, with paired gains of +2.37, +1.84 and +0.20 points. These results support using RELATE's autoregressive training scheme to learn signal tokens for record prediction. Gen and the AR readout then compare two ways to decode that learned representation, allowing deployment to use either Qwen generation or a lightweight field predictor.
Before scoring, Gen, the AR readout and the Direct readout use the same rule to fill non-severity fields whose values are uniquely determined by a setting and condition in the training data. We construct this rule from training annotations alone. For example, if every training record for outer-race fault in a setting has health state anomalous and location outer race, a prediction of outer-race fault sets these two fields to those values. Non-severity fields without a unique training value retain the model's prediction. The rule leaves the predicted condition, damage size and eccentricity unchanged.
When a benchmark group contains multiple views, we first combine their predictions into one group-level prediction. Gen selects the condition receiving the most votes. The two readouts average the candidate logits across views and select the condition with the highest mean logit. We then apply the field-filling rule using that group-level predicted condition.
The readouts restore numeric fields to the data types declared during training before scoring.
Joint EM requires the condition and every supported field to match. Field EM scores each field over its supporting groups and then averages the seven field scores equally. This puts the three methods on the same taxonomy and evaluation basis.
Revised Table 2 and Appendix G, Tables 34–36, report the common-rule record comparisons.