1. The test set is a development set, and some of its labels were revised with the method's own table
According to Section 3 (p. 4), the authors used the 201 test cases to extend the rule table from 64 to 70 rows, to develop the corpus, and to select checkpoints during development. Appendix D.2 adds that when they generated new training data aimed at the model's errors on these cases, the generation prompts kept concepts from the specific cases. As a result, earlier corpus versions contained four groups of 12–19 generated rows that paraphrase test cases. Section 7 states that the authors have no full semantic-neighbour audit for the main corpus.
The paper does not say where the original labels came from. It only states that each case "carried an original tier label" (p. 4), and two pairs of cases had identical text but opposite labels (Appendix D.3). Twenty-five labels were flagged for review against the rule table, and the authors changed 12 of them. The audit script marked the other 176 labels AGREE by default, which the authors note is "not evidence that 201 manual decisions were completed" (Appendix D.3); Section 3 should say that there is no record of a human review of these labels. A clinician, working within the authors' review process, checked the same 25 cases and agreed with 9 of the 12 changes (p. 4). So part of the reference was relabelled by the authors, using the same table that produces the training labels, which may pull it towards the method's own rules. The agreement gain is +12.4 points with the reference, +11.4 with the clinician's labels and +10.0 with the original labels (Table 31), so the changes help the method somewhat.
The only external test, Oxford-40 (Section 4.5), shows no gain in agreement. As the authors acknowledge, the main results measure agreement with a development reference. This limits what the paper can claim about the method.
2. The training data were developed with a flawed evaluator
All development comparisons between corpus versions, class weights and checkpoints in Tables 7–11 and 24 used an evaluator that read the hidden case profile and overwrote the model's verdict (Appendix B.6, pp. 19–20; Appendix C, p. 22). Some of the evaluator's trigger strings were written for specific development cases. The training corpus itself was developed under this evaluator (Appendix C.1, "adaptive data development"), and Table 6 lists model selection among the uses of these protocols. The choice of training data was therefore guided by scores that, as the authors write, "must not be interpreted as pre-heuristic model policy-reference agreement" (p. 22). This is a serious flaw in the development process. The main text says only that a post-hoc heuristic overwrote verdicts in earlier runs (pp. 4, 9). It does not say that the heuristic read the hidden profile, that it used strings written for specific cases, or that the development comparisons ran under it.
Please say this clearly in the main text, and explain how the final configuration (corpus version and number of epochs) was chosen among these variants. Table 7 notes that two conditions "have no designated operating checkpoint", but the paper does not give the selection criterion. Please also discuss whether choosing the configuration with this evaluator could bias the clean results in Table 1.
The GRPO run shows how much this can distort results: it scores 151/201 with the heuristic and 91/201 without it (Appendix F.2).
3. The main baseline is an untuned model
The main comparison model is Qwen3.5-9B-Base, a pretrained model without instruction tuning, prompted to act as a nurse. It can hold a conversation (its calls are as long as the trained model's, Table 28), though it asks less often about key facts. Its main weakness is the decision: even in the full-information condition, where it receives the whole post at the start, it classifies only 7 of the 42 emergencies as emergent (p. 6).
Still, an untuned model is an easy baseline. Some of the general-purpose models in Table 1 escalate as often as the trained model without any guideline training (Qwen3.5-397B and Kimi-K2.6 catch 31 and 32 of 42 emergencies, against 29), though they also over-triage more. It is therefore unclear whether guideline training is needed, or whether an instruction-tuned model of the same size would show the same shift.
Three baselines would settle this: an instruction-tuned 9B model of the same family, the same 9B model with the rule table in its prompt, and a prompt that simply tells the model to escalate when in doubt.
4. The agreement gain does not replicate in the second environment
In the first environment, the agreement gain holds in a second run (+11.4 points) and with a second caller (+10.4 points). In the second environment, however, the three runs differ from their base model by −5.5 to +4.0 points, and none of these differences is significant (Table 2). Two runs that differ only in their seed differ by 19 cases (p = 0.018, Section 4.3). Even the same base weights change about a quarter of their verdicts between environments (p. 4). This environment effect mixes numerical noise with a change in the caller's input (about 300 extra gateway tokens per request, Appendix I); please separate the two. Please also state why a second environment was used, and whether the number of runs and seeds was fixed before the results were seen.
The zero-shot GPT-5.5 nurse reaches 75.1% (p = 0.90 against the trained model), although GPT-5.5 also plays the caller. On Oxford-40, the trained runs agree on 24–26 of 40 vignettes, compared with 26 for the base model and 33 for zero-shot GPT-5.5 in a single-turn setting. They get only 0–2 of the 14 urgent vignettes right, compared with 7 for the base model (Table 38).
What does hold up is a shift towards escalation, and the abstract and introduction already present the result this way. This makes the missing baselines in point 3 more important, since a prompt might produce a similar shift.
5. The controls do not show that the guideline is what helps
The paper claims that the rule-derived labels teach the decision, not the questioning (Section 4.2). The decision part is reasonably well supported. The permutation controls show that the decision is learned from the final message (the tier-word control still learns the tier from its unchanged next-step sentence, p. 6), and the full-information condition shows that the untuned model fails at deciding even with all facts. However, the two matched controls rest on one or two runs each, all in the second environment. The "not the questioning" part rests only on how often three keyword-coded topics come up (Figure 2a). Whether the questions fit the case is not measured, and whether decisive facts are elicited (Table 35) is not reported for the controls.
The abstract goes further and states that "the rule-derived labels carry the escalation gain" (p. 1). The controls only show that correct labels carry it, not that the labels need to come from the guideline. Labels from another reasonable source, such as a strong LLM without the table, might work just as well. One control would test this, and a second would show the role of class weighting:
- Train with tiers from another source, for example an LLM that labels the profiles without the table, ideally on dialogues regenerated for those labels. If this works as well, the gain is not specific to the guideline.
- Train without class upweighting. Resampling raises the share of emergent cases from 21% of the dialogues to 31% of training examples (Figure 4). The second permutation control, which replaces the whole final message, has the same class counts and calls almost no emergent case emergent (pp. 49–51), so the weights alone do not explain the gain. Combined with correct labels, however, they may account for part of the shift towards escalation, including the over-triage.
This matters all the more because the "guideline" labels involve more judgment than the framing suggests, and follow a different convention from the reference:
• The generator LLM writes each profile for a tier fixed in advance and chooses which rows to cite, with only limited checks that the profile facts actually match those rows (Section 2.3, Appendix B.1).
• When two LLM reviewers apply the same table to candidate training dialogues, they agree on only 44.1% of them (κ = 0.285). Where both commit to a tier, agreement is 58.4% (κ = 0.374), and almost all of these disagreements go one way: one reviewer calls 266 dialogues emergent that the other rates lower, against 5 the other way round (Section 7; Appendix C.9, Table 17).
• The authors themselves note that the training labels follow a literal reading of the table, while the reference follows a contextual one (Appendix C.15). This mismatch appears only in the appendix and should be stated in the main text.
The authors acknowledge this in Section 7 and in Appendix C.9, where they write that the determinism claim is "about the interface, not about the semantics of applying the table" (p. 30). Please also say this in Section 2, next to the statement that every training label is "a deterministic function of named rows".
6. Limited novelty.
Each part of the method has been done before:
• Training on labels from rules that are not needed at inference is data programming and rule distillation (Ratner et al., 2016; Hu et al., 2016), both of which the paper cites.
• Synthetic triage and history-taking dialogues have been generated for training and evaluation before, for example TriageSim (Srirag et al., 2026) and Note2Chat (2026). The paper cites neither.
• The obvious alternative to distilling the guideline into weights is to use it at inference time. The paper's baselines of this kind are GPT-5.5 with the table pasted into its prompt (Table 1, Appendix D.8) and a deterministic execution of the rules (Appendices C.11, D.8). It does not compare against LLM systems that retrieve or step through a guideline, such as the cited TriageAgent (Lu et al., 2024), a multi-agent system that navigates AMA flowcharts, or the concurrent ALLMTS (NeurIPS 2026).
• Nurse telephone triage already runs on structured protocols in practice (e.g. Schmitt-Thompson). The paper should discuss how its rule table relates to such protocols.
• The evaluation methods (cost-sensitive comparison, cost curves) are established, and the paper cites them.
The paper's main value is its careful record of what can go wrong when evaluating a triage agent. It would be stronger if it were framed that way.