acceptodds
Withdrawn from ICLR 2027

Guideline-as-Oracle: Rule-Derived Labels Teach a Triage Agent When to Escalate

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Can a published clinical guideline replace per-dialogue expert labels when training an interactive triage agent? We study ophthalmic telephone triage: a guideline, compiled by hand into a 70-row rule table, labels every synthetic training dialogue with the most severe tier among the rows its caller profile cites, so no dialogue is labeled by a person. On a 201-case development reference adjudicated by the authors, fine-tuning a 9B agent raises emergent recall in simulated calls from 9.5% to 69.0% and halves under-triage, and this escalation gain holds in every evaluated setting (not fully crossed): five training runs, two software environments, two caller simulators, and five reference-robustness views, including a clinician's review of the flagged cases. The rule-derived labels carry the escalation gain: controls trained on the same dialogues with the final tier words, or entire final messages, permuted raise the same topics but call 0 and 2 of 42 emergent cases emergent, while the first control's next-step recommendations, whose training text was not permuted, still track the rule-derived tier; given the complete case, the untuned model still calls only 7 of 42 emergent cases emergent. Training shifts the balance between under- and over-triage, by amounts that vary across runs of one configuration. Agreement rises from 61.7% to 74.1% in the pre-specified comparison, but over-triage ranges from 26 to 54 calls across runs, so the agreement difference ranges from +12.4 to −5.5 points. Every run costs less than its base model once an under-triaged call counts more than 1.41 over-triaged ones, and significantly less in all five once it counts three. On 40 external vignettes the trained model escalates more but does not agree more. The main results measure agreement with a development reference, not clinical validity.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM agents, synthetic data, weak supervision, clinical triage, multi-turn dialogue, cost-sensitive evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_ZUQ3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors compile ophthalmic guidance into a rule table, generate rule-labeled synthetic dialogues, and fine-tune a triage model. Training increases emergency escalation across repeated runs, although overall agreement varies with over-triage.

Critical strengths
  • The paper reports under- and over-triage separately and acknowledges the variability across runs and software environments.
  • The final-message permutation control and full-information condition help locate the source of the escalation change. The limitations are stated candidly.
Critical weaknesses
  • The primary evaluation cannot establish generalization. The same 201 cases informed rule expansion, corpus development, and checkpoint selection. Their reference labels were partly revised using the rule table under evaluation. Alternative label views do not make these cases an independent test set.
  • The independent check does not show improved overall decisions. On Oxford-40, the trained runs agree with 24–26 of 40 labels, versus 26 for the base model. They identify all emergencies but escalate 12–13 of 14 urgent cases to emergent. This is consistent with a shift toward escalation, without evidence of improved discrimination.
  • The main comparison uses an untuned base model. Without a same-size instruction-tuned baseline or a comparator adjusted to a similar escalation rate, the results do not establish the benefit of the rule-derived construction over simpler ways of changing the operating point. The general-purpose model comparisons do not resolve this issue.
  • The supervision source is not independently validated. The 70-row table has neither a passage-level mapping to the guideline nor independent panel validation (Section 2.2). Deterministic assignment from cited row IDs verifies internal consistency, but does not establish that a profile satisfies the cited clinical trigger.
Questions for authors
  • Is there any evaluation cohort that was untouched by rule revision, corpus construction, and model selection?
  • At matched emergency recall, how does GAO-Triage compare with a same-size instruction-tuned model or a simpler escalation-adjusted baseline?
Reviewer_vb7hOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a published clinical guideline can replace expert labels when training a triage agent that talks to callers. The authors turn AAO guidance into a 70-row table by hand. Each row links a trigger to one of three urgency tiers.

LLMs write synthetic caller profiles that cite rows of the table. Each profile gets the most severe tier among its rows, and the profiles are turned into 3,000 nurse–caller dialogues, mostly by LLMs. The authors then fine-tune Qwen3.5-9B-Base on the nurse turns.

In simulated calls on 201 cases taken from Reddit posts, the model catches more emergencies: emergent recall goes from 4/42 to 29/42. Agreement with the reference rises from 61.7% to 74.1% in the pre-specified test. The gain on emergencies holds across five training runs and two software environments, but the gain in agreement appears only in the first environment.

The authors use permutation controls and a full-information condition to argue that the labels teach the model when to escalate, not what to ask. On 40 external vignettes, the trained model escalates more often but does not agree better with the reference.

Critical strengths

• The problem matters for patient safety. The paper treats under-triage and over-triage separately and reports costs by error direction, not just accuracy. • The reporting is very transparent. The paper documents negative and null results in detail, including the failed GRPO run, the boundary-pair ablation, the dependence on the software environment, and the overwritten historical runs. • The authors repeat the main experiment across five training runs, two caller simulators and two software environments. This shows run-to-run variation that a single run would hide. • The full-information condition is a good way to separate asking questions from making the decision. • The authors release the rule table, the executable matcher and per-case categorical results.

Critical weaknesses

1. The test set is a development set, and some of its labels were revised with the method's own table

According to Section 3 (p. 4), the authors used the 201 test cases to extend the rule table from 64 to 70 rows, to develop the corpus, and to select checkpoints during development. Appendix D.2 adds that when they generated new training data aimed at the model's errors on these cases, the generation prompts kept concepts from the specific cases. As a result, earlier corpus versions contained four groups of 12–19 generated rows that paraphrase test cases. Section 7 states that the authors have no full semantic-neighbour audit for the main corpus. The paper does not say where the original labels came from. It only states that each case "carried an original tier label" (p. 4), and two pairs of cases had identical text but opposite labels (Appendix D.3). Twenty-five labels were flagged for review against the rule table, and the authors changed 12 of them. The audit script marked the other 176 labels AGREE by default, which the authors note is "not evidence that 201 manual decisions were completed" (Appendix D.3); Section 3 should say that there is no record of a human review of these labels. A clinician, working within the authors' review process, checked the same 25 cases and agreed with 9 of the 12 changes (p. 4). So part of the reference was relabelled by the authors, using the same table that produces the training labels, which may pull it towards the method's own rules. The agreement gain is +12.4 points with the reference, +11.4 with the clinician's labels and +10.0 with the original labels (Table 31), so the changes help the method somewhat. The only external test, Oxford-40 (Section 4.5), shows no gain in agreement. As the authors acknowledge, the main results measure agreement with a development reference. This limits what the paper can claim about the method.

2. The training data were developed with a flawed evaluator

All development comparisons between corpus versions, class weights and checkpoints in Tables 7–11 and 24 used an evaluator that read the hidden case profile and overwrote the model's verdict (Appendix B.6, pp. 19–20; Appendix C, p. 22). Some of the evaluator's trigger strings were written for specific development cases. The training corpus itself was developed under this evaluator (Appendix C.1, "adaptive data development"), and Table 6 lists model selection among the uses of these protocols. The choice of training data was therefore guided by scores that, as the authors write, "must not be interpreted as pre-heuristic model policy-reference agreement" (p. 22). This is a serious flaw in the development process. The main text says only that a post-hoc heuristic overwrote verdicts in earlier runs (pp. 4, 9). It does not say that the heuristic read the hidden profile, that it used strings written for specific cases, or that the development comparisons ran under it. Please say this clearly in the main text, and explain how the final configuration (corpus version and number of epochs) was chosen among these variants. Table 7 notes that two conditions "have no designated operating checkpoint", but the paper does not give the selection criterion. Please also discuss whether choosing the configuration with this evaluator could bias the clean results in Table 1. The GRPO run shows how much this can distort results: it scores 151/201 with the heuristic and 91/201 without it (Appendix F.2).

3. The main baseline is an untuned model

The main comparison model is Qwen3.5-9B-Base, a pretrained model without instruction tuning, prompted to act as a nurse. It can hold a conversation (its calls are as long as the trained model's, Table 28), though it asks less often about key facts. Its main weakness is the decision: even in the full-information condition, where it receives the whole post at the start, it classifies only 7 of the 42 emergencies as emergent (p. 6). Still, an untuned model is an easy baseline. Some of the general-purpose models in Table 1 escalate as often as the trained model without any guideline training (Qwen3.5-397B and Kimi-K2.6 catch 31 and 32 of 42 emergencies, against 29), though they also over-triage more. It is therefore unclear whether guideline training is needed, or whether an instruction-tuned model of the same size would show the same shift. Three baselines would settle this: an instruction-tuned 9B model of the same family, the same 9B model with the rule table in its prompt, and a prompt that simply tells the model to escalate when in doubt.

4. The agreement gain does not replicate in the second environment

In the first environment, the agreement gain holds in a second run (+11.4 points) and with a second caller (+10.4 points). In the second environment, however, the three runs differ from their base model by −5.5 to +4.0 points, and none of these differences is significant (Table 2). Two runs that differ only in their seed differ by 19 cases (p = 0.018, Section 4.3). Even the same base weights change about a quarter of their verdicts between environments (p. 4). This environment effect mixes numerical noise with a change in the caller's input (about 300 extra gateway tokens per request, Appendix I); please separate the two. Please also state why a second environment was used, and whether the number of runs and seeds was fixed before the results were seen. The zero-shot GPT-5.5 nurse reaches 75.1% (p = 0.90 against the trained model), although GPT-5.5 also plays the caller. On Oxford-40, the trained runs agree on 24–26 of 40 vignettes, compared with 26 for the base model and 33 for zero-shot GPT-5.5 in a single-turn setting. They get only 0–2 of the 14 urgent vignettes right, compared with 7 for the base model (Table 38). What does hold up is a shift towards escalation, and the abstract and introduction already present the result this way. This makes the missing baselines in point 3 more important, since a prompt might produce a similar shift.

5. The controls do not show that the guideline is what helps

The paper claims that the rule-derived labels teach the decision, not the questioning (Section 4.2). The decision part is reasonably well supported. The permutation controls show that the decision is learned from the final message (the tier-word control still learns the tier from its unchanged next-step sentence, p. 6), and the full-information condition shows that the untuned model fails at deciding even with all facts. However, the two matched controls rest on one or two runs each, all in the second environment. The "not the questioning" part rests only on how often three keyword-coded topics come up (Figure 2a). Whether the questions fit the case is not measured, and whether decisive facts are elicited (Table 35) is not reported for the controls. The abstract goes further and states that "the rule-derived labels carry the escalation gain" (p. 1). The controls only show that correct labels carry it, not that the labels need to come from the guideline. Labels from another reasonable source, such as a strong LLM without the table, might work just as well. One control would test this, and a second would show the role of class weighting:

  1. Train with tiers from another source, for example an LLM that labels the profiles without the table, ideally on dialogues regenerated for those labels. If this works as well, the gain is not specific to the guideline.
  2. Train without class upweighting. Resampling raises the share of emergent cases from 21% of the dialogues to 31% of training examples (Figure 4). The second permutation control, which replaces the whole final message, has the same class counts and calls almost no emergent case emergent (pp. 49–51), so the weights alone do not explain the gain. Combined with correct labels, however, they may account for part of the shift towards escalation, including the over-triage.

This matters all the more because the "guideline" labels involve more judgment than the framing suggests, and follow a different convention from the reference:
• The generator LLM writes each profile for a tier fixed in advance and chooses which rows to cite, with only limited checks that the profile facts actually match those rows (Section 2.3, Appendix B.1).
• When two LLM reviewers apply the same table to candidate training dialogues, they agree on only 44.1% of them (κ = 0.285). Where both commit to a tier, agreement is 58.4% (κ = 0.374), and almost all of these disagreements go one way: one reviewer calls 266 dialogues emergent that the other rates lower, against 5 the other way round (Section 7; Appendix C.9, Table 17).
• The authors themselves note that the training labels follow a literal reading of the table, while the reference follows a contextual one (Appendix C.15). This mismatch appears only in the appendix and should be stated in the main text.
The authors acknowledge this in Section 7 and in Appendix C.9, where they write that the determinism claim is "about the interface, not about the semantics of applying the table" (p. 30). Please also say this in Section 2, next to the statement that every training label is "a deterministic function of named rows".

6. Limited novelty.
Each part of the method has been done before:
• Training on labels from rules that are not needed at inference is data programming and rule distillation (Ratner et al., 2016; Hu et al., 2016), both of which the paper cites.
• Synthetic triage and history-taking dialogues have been generated for training and evaluation before, for example TriageSim (Srirag et al., 2026) and Note2Chat (2026). The paper cites neither.
• The obvious alternative to distilling the guideline into weights is to use it at inference time. The paper's baselines of this kind are GPT-5.5 with the table pasted into its prompt (Table 1, Appendix D.8) and a deterministic execution of the rules (Appendices C.11, D.8). It does not compare against LLM systems that retrieve or step through a guideline, such as the cited TriageAgent (Lu et al., 2024), a multi-agent system that navigates AMA flowcharts, or the concurrent ALLMTS (NeurIPS 2026).
• Nurse telephone triage already runs on structured protocols in practice (e.g. Schmitt-Thompson). The paper should discuss how its rule table relates to such protocols.
• The evaluation methods (cost-sensitive comparison, cost curves) are established, and the paper cites them.

The paper's main value is its careful record of what can go wrong when evaluating a triage agent. It would be stronger if it were framed that way.

Questions for authors

Minor points:

• Figure 1 presents eight construction strategies. According to Table 23, only the first, assigning each label from the cited rows, is supported: the pre-specified ablation of the boundary pairs (strategy 2) detected no effect, and the other six were never tested on their own, were changed together with other things, or have only development evidence. Figure 1 should mark which strategies are supported.
• The original fine-tuned weights were deleted (Appendix F.2). Historical training seeds, complete decoding settings for some generator sources, and the Protocol B′ validator rubric were not kept (Appendices A, B.4 and F.3). Beyond the released per-case labels, this makes the work hard to reproduce.
• With full information, the trained runs still take about ten nurse turns and over-triage 37–58 calls, compared with 8 for the base model (Table 36 and Appendix I). The paper should discuss whether this is a side effect of training.
• Please print the exact evaluation prompts: the generic nurse prompt used at evaluation, any prompts that differ for the base model or the general-purpose models, and the caller-simulator prompt. Appendix B.2 shows only a family-specific training prompt, although the Reproducibility Statement says Appendix B specifies the prompts.

Questions for the authors

  1. Who assigned the original labels of the 201 cases, and how?
  2. When were the 12 labels changed, relative to the first model runs on these cases?
  3. What criterion did the audit script use to flag the 25 labels (Appendix D.3)? Apart from the two LLM auditors, did any person check the other 176 labels?
  4. How was the final configuration (boundary-revised corpus, three epochs) chosen among the development variants in Tables 7–11 and 24?
  5. Under the clean protocol, how do these baselines perform: an instruction-tuned 9B model, the 9B model with the rule table in its prompt, and a simple "escalate when in doubt" prompt?
  6. On the main corpus and under the clean protocol, how does a model perform when trained on tiers from a source other than the guideline, and when trained without class upweighting?
  7. Apart from Oxford-40, is there any evaluation set that did not inform development?
  8. On Oxford-40, the trained runs get almost none of the urgent vignettes right, while on the 201 cases their urgent recall is similar to the base model's (Table 36). What explains the difference?
Reviewer_7JmYOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary
  • The authors generate synthetic patient profiles and assign urgency labels using a manually constructed table of 70 rules informed by AAO guidelines. They then generate 3,000 synthetic ophthalmic triage dialogues, by using LLMs to simulate interactions between patients and nurses.
  • The authors fine-tune Qwen3.5-9B-Base on the nurse turns in these dialogues, using 3,998 training examples after upsampling. The fine-tuned model asks questions, assigns one of three urgency labels (Emergent, Urgent, or Routine), and provides a rationale and a care recommendation.
  • Across 201 patient cases, the fine-tuned model achieves higher recall for emergent cases than the untuned base model, but overall agreement with the reference labels does not consistently improve. When given the full information in 40 external clinical vignettes, the trained models assign higher urgency levels more often without improving overall agreement.
Critical strengths
  • The fine-tuned model identifies more emergency cases than the base model. The control experiments suggest that learning the final triage decisions contributes to this improvement.
  • The released table of 70 AAO-informed rules could be a useful resource for researchers developing guideline-based triage systems.
Critical weaknesses

I have three main concerns about the evaluation, data construction, and experiments.

  • Evaluation: The paper does not explain who assigned the original urgency labels or what criteria they used. A clinician reviewed only 25 of the 201 cases (12.4%). Because these labels determine which predictions count as correct, readers need enough information to assess their reliability. The authors also used the evaluation cases to develop the rules and training data, making it difficult to determine whether the reported improvement would generalize to new cases.
  • Data construction: It is unclear how the authors derived the 70 rules from the AAO guidelines or how they verified the resulting training labels. The paper does not identify the guideline passages that support each rule. It also does not report a review of a representative sample of the final 3,000 dialogues to assess whether the information each patient provided before the nurse’s decision supported the assigned urgency label. The reliability of the training labels therefore remains uncertain.
  • Experiments: The comparison with the untuned base model shows that supervised fine-tuning improves recall for emergency cases. To determine whether the proposed data construction method offers an advantage over standard SFT on synthetic dialogues, the authors should compare the two approaches using the same model and training budget. The label-shuffling controls show that pairing dialogues with the correct labels matters, but they leave open whether the proposed rules produce better labels than alternative methods. The paper also does not explain why GPT-5.5 performs worse when given the rule table or report results from a comparable rule-prompting experiment with the 9B model.

I also found the paper presentation is difficult to follow, particularly where it describes how the rules were developed and how the training and evaluation cases were used.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.