acceptodds
Withdrawn from ICLR 2027

Beyond Presentation: Toward Format-Robust Medical Evidence Reasoning in LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Medical evidence-source preference, where LLMs systematically favor certain evidence sources when reasoning about competing diagnoses, is an underexplored factor in reliable medical reasoning. However, whether such preferences reflect evidential content or are confounded by evidence presentation remains unclear. To investigate this question, we build PMC-ConForm, a benchmark of medical evidence conflicts from PMC-Patients case reports where evidence source and presentation format vary independently. Through detailed empirical analyses, we find that LLMs exhibit systematic medical source preferences, yet these preferences can shift substantially and even reverse under content-preserving format changes, indicating that evidence prioritization partly reflects presentation rather than evidence itself. To address this issue, we propose FORMEDER, a cross-format consensus self-distillation framework for format-robust medical evidence reasoning. We re-render the same evidence into multiple content-preserving formats and aggregate the frozen model's predictions into a soft consensus target shared across presentations. We then distill the model toward this target under each format while introducing a contraction objective to suppress residual format-dependent variation. The framework requires no external annotation and uses a single presentation at inference. Extensive experiments on eight LLMs demonstrate that FORMEDER substantially mitigates format-induced preference shifts and enhances cross-format choice consistency while faithfully preserving presentation-invariant judgments, and generalizes to unseen presentation formats.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Robustness, Medical Reasoning, Representation Sensitivity, Self-Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xyf4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper tries to probe the causal factors of the evidence preference of LLMs in medical situations. It intends to probe why LLMs prefer a certain set of evidence. For this, the authors propose the PMC-ConForm dataset which are kind of based on "virtual patients" synthesied from two patients.

Critical strengths

The general idea to improve the stability in medical AI and to reduce bias is valid. And it's also very important.

Critical weaknesses

Low Probability Coincidence in the Reported Data

It seems that the numerical values in this paper can be recalculated from the percentages. This is of extremely low probability due to rounding. The rounding function is not an injective mapping, it has no inverse mapping and breaks the multiplicative group.

Take the first line of Table 9 as an example. S0=6.25,Sours=2.63,r=0.42S_0 = 6.25, S_{ours} = 2.63, r=0.42. 6.25×0.42=2.625≈2.636.25 \times 0.42 = 2.625 \approx 2.63 (2.625 rounds to 2.63 in Microsoft Excel, to 2.62 in R or Python. Manually it's usually 2.63 since rounds half up is the standard in K-12.). So it recalculates to 2.63. This is of low probability because 2.606.25=0.416≈0.42\frac{2.60}{6.25} = 0.416 \approx 0.42 and 2.656.25=0.424≈0.42\frac{2.65}{6.25} = 0.424 \approx 0.42. So with S0=6.25S_0 = 6.25, for SoursS_{ours} there are 6 possible values, 2.60, 2.61, 2.62, 2.63, 2.64, 2.65, satisfying ⌊102×SourS0+0.5⌋102=0.42\frac{\lfloor 10^2\times \frac{S_{our}}{S_0} + 0.5 \rfloor}{10^2} = 0.42. The coincidence that the measured value can be accurately recalculated from the percentage is only 16\frac{1}{6}.

This coincidence is found for all 8 rows in Table 9. All the measured SoursS_{ours} are accurately recalculated from the rounded percentage. p=16×113×14×110×11×12×14×117≈2.3×10−6p = \frac{1}{6} \times \frac{1}{13} \times \frac{1}{4} \times \frac{1}{10} \times \frac{1}{1} \times \frac{1}{2} \times \frac{1}{4} \times \frac{1}{17} \approx 2.3 \times 10^{-6}.

Similar patterns are found in Table 1 and Table 10, where the FPS-nats-Frozen-unseen values of all models can be calculated from the seen values by multiplying by an integer percentage coincidentally.

ModelTable 1Table 10Multiplier
MedGemma-4B1.71311.9187112%
Gemma3-4B2.99763.5372118%
HuatuoGPT-o1-7B0.81780.8914109%
Qwen2.5-7B2.24002.5760115%
BioMistral-7B0.26560.3240122%
Mistral-7B0.38670.4486116%
Llama3-8B0.78830.9775124%
Qwen2.5-14B3.65254.1273113%

Take MedGemma-4B as an example: 1.7131×1.12=1.918672≈1.91871.7131 \times 1.12 = 1.918672 \approx 1.9187. This is also of very low probability because 1.7131×1.13=1.935803≈1.9358,1.7131×1.11=1.901541≈1.90151.7131\times 1.13=1.935803\approx 1.9358, 1.7131\times 1.11=1.901541\approx 1.9015. So 1.9187 is the only four-decimal number in (1.9016,1.9357)(1.9016, 1.9357) that would lead to this coincidence.

Shrink Lines Are Not Calculated from the raw data

In section F.2, the authors define that:

let $r = \frac{S_{ours}}{S_0}. The matched shrink line is rFPS0r\text{FPS}_0.

Hence we have SL=SoursS0×FPS0\text{SL} = \frac{S_{ours}}{S_0} \times \text{FPS}_0. Take MedGemma-4B as an example: (2.63/6.25)×1.7131=0.7209(2.63/6.25)\times 1.7131 = 0.7209, which is pretty far from the reported 0.7195. However, if we use the 100r100r, we have 0.42×1.7131=0.7195020.42 \times 1.7131 = 0.719502. So it seems that the shrink lines are calculated from the percentage instead of the raw data. This is not standard for science, you should calculate all numerical values with the Double type and only round them before writing the paper to prevent the amplification of the rouding errors.

Research Problem is Invalid - Part I

The problems of medical diagnosis have ground truth values. These ground truth values are not hard to get. In this case, the reason for adopting the invariant goal instead of the ground-truth accuracies is strange. IMO, it's not supported in this submission.

Take an autonomous driving algorithm as an example. For a situation that the car encountered a red light, it's clearly required by the law that the car should stop. So, in this case, there is a ground truth for the autonomous driving algorithm, and it should brake. For this situation, we simply aim to improve the decision accuracy. There's no need to improve the invariance or stability. Actually, it's the logical implication of accuracy. If the car always makes the correct decision, it will be stable. Whereas, in the situation where our car is behind a slow-moving car and an overtaking decision is needed, there's no universal ground truth on whether the algorithm should overtake or not. In this case, it's extremely hard to get a ground truth label, and we have to pursue the stability of the algorithm such that, in similar situations, it is expected to have similar behaviors.

I'm not claiming that stability against input formats is not important. However, we can simply investigate this stability against real instances with real ground truth labels. We can study whether the accuracies drop when we alter the format. Also, if the accuracy is high enough, the stability is implied. I don't see any reason to abandon the ground truth instances and pursue the virtual ones.

Moreover, when a specific format causes incorrect diagnoses, our priority should be fixing correctness, not consistency, since consistency can be artificially inflated by making correct diagnoses wrong. Practically, if a format improves accuracy, we can simply use prompt engineering to guide physicians to adopt it. Consider a calculator on Windows, Mac, or Linux: it ignores inputs like 'o,n,e,p,l,u,s,o,n,e.' While returning 't,w,o' would be nice, the lack of support for this format does not stop it from being an excellent calculator.

Research Problem is Invalid - Part II

Medical diagnosis is essentially computing quotients of sets. The essence is to identify subsets from the universal set based on certain evidence. That's by definition. For an arbitrary universal set, there are multiple partitions and each partition can be correlated with certain evidences. For instance, for Z\mathbb{Z}, one "diagnosis" is to diagnosis whether a number is odd or even. This partition is the quotient set Z/2Z={0+2Z,1+2Z}\mathbb{Z}/2\mathbb{Z} = \{0+2\mathbb{Z}, 1+2\mathbb{Z}\}. The evidence used for this diagnosis is:

π2:Z⟶Z/2Z,n⟼n mod 2.\pi_2 : \mathbb{Z} \longrightarrow \mathbb{Z}/2\mathbb{Z}, \qquad n \longmapsto n \bmod 2.

However, if the diagnosis is about whether a number is divisible by 3, the partition becomes Z/3Z={0+3Z,1+3Z,2+3Z}\mathbb{Z}/3\mathbb{Z} = \{0+3\mathbb{Z}, 1+3\mathbb{Z}, 2+3\mathbb{Z}\} and the evidence you need becomes

π3:Z→Z/3Z,n⟼n mod 3.\pi_3 : \mathbb{Z} \to \mathbb{Z}/3\mathbb{Z}, \qquad n \longmapsto n \bmod 3.

There's no unified "evidence" for medical diagnosis. Different evidence is needed for different subsets. Overlapping subsets exist. For instance, Smal de ojo∩Sconjuntivitis≠∅S_{\text{mal de ojo}} \cap S_{\text{conjuntivitis}} \neq \emptyset and they rely on different evidence πmal de ojo≠πconjuntivitis\pi_{\text{mal de ojo}} \neq \pi_{\text{conjuntivitis}}. The whole diagnosis system is not a set partition, as it doesn't adhere to the definition P=U/∼\mathcal{P} = U/\sim, (∀S∈P,S≠∅)∧(S1∪S2∪⋯=U)∧(∀i≠j,i,j∈[1,∣P∣]Si∩Sj=∅).(\forall S \in \mathcal{P}, S \neq \emptyset) \land (S_1 \cup S_2 \cup \cdots = U) \land (\forall_{i\neq j, i,j\in [1, |\mathcal{P}|]} S_i \cap S_j = \emptyset). It's the diagnosis of each dieases the partitions. And they each has their own quotient mapping which by definition indicate the evidence.

A condition qualifies as a disease precisely because specific evidence (the quotient map) allows us to stably partition patients into a quotient set, where those within the same equivalence class share similar treatment approaches. Thus, the quotient set (the disease) is induced from the quotient map (the evidence).

If the evidence were left undefined, merely showing a statistical correlation with the disease, then (as implied by the limitations section of this paper) an unmeasured confounding variable would have been overlooked, constituting a limitation of the study. However, I argue that this is not the case here. Rather, the definition of the disease itself provides the very evidence it requires. The quotient sets correspondes to the quotient maps: (π↦X/∼=im π)∧(X/∼↦π:x↦[x])(\pi \mapsto X/\sim = \mathrm{im}\, \pi) \land (X/\sim \mapsto \pi: x \mapsto [x]). This circularity in the evidence casts doubt on the validity of the entire research question. It's somewhat by definition. The whole research question is somewhat tautological. For instance, an AIDS patient is an AIDS patient because it's defined as πAIDS:U⟼E\pi_{\mathrm{AIDS}}: U \longmapsto E, E={HIV+,HIV−}E=\{\mathrm{HIV}^{+},\mathrm{HIV}^{-}\}, a∼AIDSb:⇔πAIDS(a)=πAIDS(b)a \sim_{\mathrm{AIDS}} b :\Leftrightarrow \pi_{\mathrm{AIDS}}(a)=\pi_{\mathrm{AIDS}}(b). Hence the definition of AIDS requires microbiology as this is inherently defined in πAIDS\pi_{AIDS}.

Diseases DD for which πD\pi_{D} are not defined on pathology and microbiology evidence but instead on physical exam, physiology, and self-report include Myopia, Color Vision Deficiency, Pes Planus, Hypertension, Malocclusion, Erectile Dysfunction, Premature Ejaculation, etc.

Model Obsolescence

The models used in this paper include Qwen 2.5, MedGemma, and Mistral 7B. They are mostly old models released between 2023 and 2025.

These models all predate or are contemporary to DeepSeek-R1. From a technical standpoint, they all seem to be pre-R1 models. Nowadays, most people are using post-R1 models that are able to think with those super long reasoning chains. I doubt whether the format will have this much influence on those post-R1 models because these modern models will repeat the information in natural language and reason about each of your provided records carefully. For difficult diagnoses, they can reason for tens of thousands of tokens. Will they still be this sensitive to format after this long reasoning chain? I think the authors have to conduct experiments with post-R1 models to demonstrate the reported findings.

I also find the following sentence in the prompt (page 17):

Read the records and answer with a single letter. Do not explain your reasoning.

So it seems that the authors are actually asking the models not to think. Bad experimental design, in my opinion. Nowadays, almost all models utlizes GRPO and can produce extra-long reasoning behaviors. These old models that cannot reliably generate long chains of thought have largely been phased out.

Furthermore, long reasoning has significantly improved the performance of LLMs across most tasks. By 2026, almost all new models are those with long reasoning. In the context of medical diagnosis, enhancing diagnostic performance is clearly the top priority. I note that the training method proposed in this paper is inherently suited for models without reasoning processes. If one were to actively abandon long chain-of-thought techniques, which could substantially boost performance, in pursuit of this so-called stability, I would consider that a case of putting the cart before the horse.

Questions for authors

Why are the seeds used for experiments 20260908 and 20260915? Are they dates?

Why do the experiments in the Motivation section use the larger seed 20260915 while the experiments in the Experiments section use the 20260908 one?

Why not give more examples?

9 comments

Dear Reviewer xyf4, we thank you for the detailed comments. Below, we address each of your concerns.

Weakness 1 & 2

A: Thank you for your careful review! We immediately examined the relevant records on our local computers and servers and identified a problem with the handling of intermediate precision in our agent-assisted result-processing workflow: these entries were derived from rounded values rather than computed from the full-precision results. The AI Use Statement of the revised manuscript describes the role of these agents in more detail.

(1) Cause. The agent that processed the evaluation outputs saved the full-precision results in a temporary directory outside our project directory. The table-generation agent could not locate them and used the rounded values of a draft summary as exact inputs: 100r100r rounded to an integer (the 100r100r column of Tab. 9) and the frozen model's unseen/seen FPS ratio rounded to two decimals. It then derived the entries in Table 1 through Sours=rS0S_{\mathrm{ours}}=rS_0 and FPS0,unseen=FPS0,seen×\mathrm{FPS_{0,unseen}}=\mathrm{FPS_{0,seen}}\times this ratio, carrying the rounding errors into them.

Table 1. Affected entries (r~=round(100r)/100\tilde r=\mathrm{round}(100r)/100).

EntryHow it was computed in the submissionCorrected
Tab. 9, SoursS_{\mathrm{ours}}S0S_0 (two decimals) ×\times r~\tilde r, rounded half up to two decimalsrecovered original value
Tab. 9, shrink liner~×FPS0\tilde r\times\mathrm{FPS_0} (Tab. 1 of the paper, four decimals)r⋅FPS0r\cdot\mathrm{FPS_0}, with r=Sours/S0r=S_{\mathrm{ours}}/S_0 at full precision
Tab. 9, r⋅FPS0/FPSoursr\cdot\mathrm{FPS_0}/\mathrm{FPS_{ours}}rounded shrink line // FPSours\mathrm{FPS_{ours}} (Tab. 1 of the paper, four decimals)computed at full precision
Tab. 10, Frozen FPS (unseen formats)FPS0\mathrm{FPS_0} (Tab. 1 of the paper, four decimals) ×\times unseen/seen ratio rounded to two decimalsrecovered original value

(2) Original values. We manually recovered the original values from this temporary directory, as listed below.

Table 2. Unrounded SS and FPS values (single precision).

BackboneS0S_0SoursS_{\mathrm{ours}}FPS0,seen\mathrm{FPS_{0,seen}}FPSours,seen\mathrm{FPS_{ours,seen}}FPS0,unseen\mathrm{FPS_{0,unseen}}
MedGemma-4B6.24645422.62076881.71314760.076659431.9263515
Gemma3-4B11.6422564.1792032.99757120.076931273.5335312
HuatuoGPT-o1-7B2.8897151.68452160.817778770.0335710050.8948341
Qwen2.5-7B9.3645053.81065632.23998020.073574622.5813103
BioMistral-7B0.94433150.512268360.265610220.024821830.32441524
Mistral-7B1.5702180.74638330.386698070.0311776140.44773838
Llama3-8B2.84462451.4678990.78834450.0601686720.97918934
Qwen2.5-14B15.786975.2322243.65250920.120626914.113172

Applying the steps in Table 1 to these values reproduces the submitted entries.

(3) Impact. Among the 32 affected entries, the largest relative difference is 0.97%. In your MedGemma-4B example, Sours=2.6208S_{\mathrm{ours}}=2.6208 rather than 2.63, and the shrink line is (2.6207688/6.2464542)×1.7131476=0.7188(2.6207688/6.2464542)\times1.7131476=0.7188 rather than 0.7195. No conclusion changes: FORMEDER retains 33%–58% of frozen prose strength, its FPS remains 5.8–14.2× below the matched shrink line, and every backbone still has lower unseen-format FPS than its corrected frozen reference in Tab. 10.

(4) Revision. We have recomputed Tab. 9 and the corresponding range in App. F.2, and replaced the frozen unseen-format FPS in Tab. 10. All derived quantities are now computed at full precision and rounded only for display, and Tab. 9 reports S0S_0 and SoursS_{\mathrm{ours}} to four decimals so that rr and the shrink line can be recomputed from the table. We also fixed the RA aggregation. In some evaluation runs, one single-record reply was missing, and the aggregation excluded it instead of scoring it as an error, so these runs used a denominator of 5,287 rather than 5,288. All RA values now use the same denominator of 5,288 (App. F.1), which lowers the affected entries in Tab. 1 and Tab. 2 of the paper by 0.01, with rankings and conclusions unchanged. All tables in this response use corrected values.

We fully recognize the importance of numerical consistency and reproducibility in empirical research, and we sincerely apologize for these errors in the submitted tables. Thank you again for helping us identify and correct this issue.

Weakness 3

A: Thank you for this thoughtful comment! We agree that different diseases are defined by different evidence. For this reason, our paper does not judge which evidence a diagnosis should rely on, but studies whether a model's preference changes when only the presentation of the same evidence changes.

(1) This question is well-posed whichever evidence defines a disease. Each comparison holds the evidence, candidate diagnoses, and positions fixed and changes only the layout of the records, so the patient and the definition of each disease stay the same. In your notation, πAIDS\pi_{\mathrm{AIDS}} takes the same value whether the HIV test result is written as a sentence or as a JSON field, and so does πhypertension\pi_{\mathrm{hypertension}} whether blood pressure measurements appear in prose or in a table. The appropriate answer should therefore not depend on the layout. Yet for 9.8%–17.1% of conflicts, some layout change reverses the frozen model's preference (Sec. 3).

(2) The source ranking motivates this question but is not its premise. Observation 1 shows that models weigh evidence sources systematically, which raises the question of whether this weighting depends on presentation. The ranking describes how models weigh sources on PMC-ConForm rather than a clinical evidence hierarchy, and we have made this scope explicit in Observation 1 and the abstract. As you point out, case reports may over-represent diagnoses confirmed by pathology or microbiology, so the ranking may differ for conditions diagnosed mainly by physical examination or physiological measurement. We now note this in the Limitations. Neither our metrics nor FORMEDER uses this ranking: FORMEDER's target is each model's own cross-format consensus for each instance.

Weakness 4

A: Thank you for this insightful comment! It raises an important question for practical use: whether format sensitivity persists under long reasoning.

We originally chose these models for two reasons. First, our preference score is computed from answer probabilities, and FORMEDER fine-tunes the model with LoRA, so both require open-weight models. Second, to test whether medical adaptation or model size improves format robustness, we paired three medical models with their general-purpose base models (MedGemma-4B/Gemma3-4B, HuatuoGPT-o1-7B/Qwen2.5-7B, BioMistral-7B/Mistral-7B) and added Llama3-8B and Qwen2.5-14B. When we selected the models, MedGemma-4B/Gemma3-4B was the most recent such pair. Neither medical adaptation nor model size substantially raises frozen FCC (Tab. 1 of the paper).

We would like to clarify that the single-letter instruction was introduced for reliable scoring rather than to suppress reasoning. In preliminary runs without this instruction, models often explained first and did not always end with a clear option letter, so their answers could not be parsed consistently across models and formats. In this direct-answer setting, a single letter at the first answer position also provides the option-letter probabilities used by the preference score, and greedy decoding gives a deterministic score for each input (App. B.4).

To address your concern, we further enabled reasoning for HuatuoGPT-o1-7B, a medical model trained for complex reasoning, and Qwen3-8B, a post-R1 model, using the same prompts on the full test set. With reasoning enabled, the single-letter instruction only constrains the final answer, and both models still reason before giving it. Each model uses its own generation configuration, so Qwen3-8B uses sampling in this mode, as its developers recommend against greedy decoding with reasoning. The results are as follows.

Table 3. Main metrics without and with reasoning.

ModelReasoningFPS↓FCC↑ (%)RA↑ (%)
HuatuoGPT-o1-7Boff0.817881.8340.62
HuatuoGPT-o1-7B + FORMEDERoff0.033693.8643.32
HuatuoGPT-o1-7Bon2.121081.6656.09
HuatuoGPT-o1-7B + FORMEDERon0.246388.8759.02
Qwen3-8Boff2.181888.6828.99
Qwen3-8B + FORMEDERoff0.084195.7431.62
Qwen3-8Bon2.677479.5035.31
Qwen3-8B + FORMEDERon0.318888.5737.05

As shown, reasoning improves accuracy but does not remove format sensitivity. With reasoning, frozen FCC is 81.66% for HuatuoGPT-o1-7B and 79.50% for Qwen3-8B, no higher than without reasoning. A likely reason is that, even when a model restates the records in its reasoning, which details it restates and how it weighs them can still depend on how the records are presented. FORMEDER remains effective in this setting. Although trained without reasoning, it raises FCC to 88.87% and 88.57% and improves RA for both models. This suggests that FORMEDER reduces how much the model's judgment depends on the format, so the effect carries over when the model reasons before answering. We have added these results to the revised manuscript.

Question 1 & 2

A: Yes, both seeds are dates in YYYYMMDD form. Before submission, we had an LLM-based agent rerun our experiments with fixed seeds to ensure reproducibility, and the agent chose these dates as the seeds. 20260908 is the training seed for all trained models (Tab. 8), and 20260915 is the resampling seed for the bootstrap intervals in the analyses (App. C.1). The Motivation section analyzes frozen models without training, so only the resampling seed appears there.

Question 3

A: Thank you for this helpful suggestion! In the revised manuscript, we have replaced the original case study with five complete examples from the PMC-ConForm test set: two conflicts with a format-induced reversal, two without, and one failure case of FORMEDER. Each shows the full records in prose and in a structured format, with the answers and preference scores of MedGemma-4B before and after FORMEDER under all four position configurations.

We hope our responses clarify the concerns raised. We have revised the manuscript accordingly. Thank you again for your careful review!

Reviewer_a9atOpenReview ↗
Rating
1(−2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies presentation sensitivity in medical evidence reasoning. The authors introduce a benchmark called PMC-ConForm. Evidence appears in five different presentation formats. These include prose and JSON. Other formats include tables and structured text. Eight LLMs demonstrate systematic medical evidence preferences. However, presentation changes can reverse individual diagnostic preferences.

The authors propose ForMedER to address this problem. It uses consensus-based self-distillation. Four presentations generate a shared training target. Experiments show substantial improvements in format consistency. The results also demonstrate generalization to unseen formats.

Critical strengths

The paper identifies a meaningful problem. Its proposed optimization method is technically coherent. The mathematical results appear correct.

Critical weaknesses

I havr several concerns weaken the experimental conclusions. The most important concern involves a missing baseline.

Three concerns drive my decision:

  • The baseline comparisons do not include simple inference-time alternatives. Explicit format-invariance prompting and prompt-based canonicalization could improve consistency without additional training. Also, since the formats are deterministically reversible, canonicalizing inputs to prose could achieve perfect format consistency. The paper does not establish why ForMedER is preferable.
  • Cross-format self-distillation combined with consistency regularization is conceptually close to existing techniques. Stronger consistency-learning baselines are necessary.
  • The benchmark combines evidence from different patients without a correct joint diagnosis. Improved consistency therefore does not establish improved medical reasoning.

M1. Missing Canonicalization Baseline

The benchmark uses deterministic rendering functions. These functions preserve every original evidence sentence. They also preserve sentence order.

Known renderers can reconstruct the original prose. Or can we already solve this issue with proper prompting? This raises a fundamental question. Why train a model for deterministically reversible transformations?

Consider this alternative approach:

  1. Convert structured evidence into prose.
  2. Feed the reconstructed prose into the frozen model.
  3. Obtain the diagnostic prediction.

Every presentation would produce identical model inputs. Deterministic inference would therefore produce identical predictions. This approach requires no model training. It also requires only one LLM inference. Alternatively, we can emphasize the problem in the prompt, and this may help solve it. These should at least exist as baselines.

M2. Cross-Format Consensus May Not Represent Shared Judgment

Averaging predictions reveals the judgment shared across presentations. The consensus target is mathematically valid. However, its interpretation requires additional justification.

Stability Does Not Imply Correctness

Suppose every presentation produces the same incorrect diagnosis. Their consensus will also favor that diagnosis. The resulting prediction will be perfectly consistent. However, its medical reasoning remains incorrect. Therefore, cross-format consensus does not guarantee clinical correctness. It only aggregates the model's original predictions.

Equal Averaging Can Produce Unexpected Preferences

Consider four presentations of the same evidence. Three presentations prefer diagnosis B. However, the arithmetic consensus can strongly prefer diagnosis A. This happens despite three presentations favoring B. It does not guarantee majority preference preservation. The problem involves unequal answer probability mass. One presentation can dominate the pooled distribution. Equal averaging does not prevent this behavior.

M3. Synthetic Conflicts Do Not Establish Clinical Validity

The benchmark combines records from different patients. Each patient originally supports a different diagnosis. The constructed conflict presents these records together. However, the prompt describes them as one patient. This creates an artificial clinical situation. The combined evidence has no established correct diagnosis.

Consequently, I think the experiment cannot measure correct conflict resolution. It can measure preference changes under format interventions. I think these objectives are fundamentally different.

Another concern involves evidence informativeness. Different sources naturally contain different diagnostic information. For example, pathology may contain highly specific findings. Patient reports may contain nonspecific symptoms. Preferring pathology does not necessarily indicate source bias. It may reflect genuine differences in diagnostic evidence.

M4. Consensus Stability May Be Misleading

The authors show that averaging more presentations reduces disagreement. But isn't this behavior already mathematically expected? The subsets increasingly overlap as their size grows. At 13 presentations, both subsets become identical. Their disagreement must therefore equal zero.

Also, averaging predictions generally reduces variance. This does not necessarily make the consensus more reliable. Does this experiment really justify the proposed consensus target?

M5. Incomplete Evaluation of Mixed Presentation Formats

The paper considers five different formats. However, only 13 of 25 possible combinations are evaluated. Why exclude the remaining 12 combinations?

For example, JSON and table combinations are missing. Real medical records often contain mixed formats. Would the method still work in these settings?

M6. Missing Strong Consistency-Training Baselines

I think the current baselines are insufficient. Per-rendering KD starts from its own frozen teacher. Its KL loss is zero at initialization. Therefore, it essentially reproduces the frozen model.

The other baselines do not directly optimize cross-format consistency. But isn't consistency the main objective of ForMedER?

Why not compare against symmetric KL or logit pairing? For example:

LKL=DKL(p1∥p2)+DKL(p2∥p1)L_{\mathrm{KL}} = D_{\mathrm{KL}}(p_1 \Vert p_2) + D_{\mathrm{KL}}(p_2 \Vert p_1)

M7. Missing Statistical Uncertainty

The experiments use only one reported training seed. The main results also lack confidence intervals. Some RA improvements are relatively small. Are these improvements statistically significant?

Questions for authors

The suggested fixes corresponding to the concerns above are:

M1. Evaluate deterministic canonicalization followed by frozen inference. Also evaluate normalization using an LLM. The authors should establish advantages beyond reversible serialization.

M2. Compare different aggregation methods. Is there a justification for the chosen method? Possible alternatives include conditional probability normalization and averaging diagnostic log odds.

M3. Ask clinicians to evaluate competing diagnostic evidence. Compare model preferences against these assessments. Alternatively, narrow the paper's central claims.

M4. Compare consensus predictions using disjoint groups of formats.

M5. Explain why the remaining 12 format combinations were excluded and whether the method works with mixed formats such as JSON and tables.

M6. Compare against established consistency-regularization methods. Use identical training data and comparable computational budgets.

M7. Can you share confidence intervals for the main results?

4 comments

Dear Reviewer a9at, we thank you for the thorough and constructive comments. Below, we address each of your concerns in detail.

M1

"Known renderers can reconstruct the original prose. Or can we already solve this issue with proper prompting? ... Why train a model for deterministically reversible transformations? ... These should at least exist as baselines."

A: Good point! For a format with a known inverse, canonicalization is indeed the simplest fix. Our renderers are invertible only because they were designed for measurement: this guarantees that each format change preserves the evidence exactly, so that its effect can be measured in isolation. However, in deployment, records arrive in layouts produced by many different systems, and no inverse is known in advance. We therefore train the model itself to make the same judgment whatever layout it receives, including layouts unseen in training. Following your suggestion, we compare FORMEDER with three inference-time baselines on the frozen models: deterministic canonicalization (inverting the four seen structured formats back to prose), a format-invariance prompt, and LLM normalization (the same model rewriting each structured record into prose). All methods are evaluated on both seen and unseen formats.

Table 1. Inference-time baselines averaged over eight backbones.

MethodFPS↓, seenFCC↑ (%), seenRA↑ (%), seenFPS↓, unseenFCC↑ (%), unseenRA↑ (%), unseen
Frozen1.607788.4645.371.850182.4742.14
Canonicalization0.0000100.0055.421.850182.4742.14
Format-invariance prompt1.661588.6445.041.900482.6642.45
LLM normalization, same model1.055290.5447.301.381983.9143.25
FORMEDER0.062296.2649.790.241291.5245.69

(1) Canonicalization works only for formats with a known inverse. On the four seen formats, it maps every presentation back to prose, so format sensitivity disappears by construction (FPS 0, FCC 100%) and RA equals prose accuracy. However, on the four unseen formats, which we hold out to simulate layouts without a known inverse, canonicalization changes nothing, while FORMEDER still reduces FPS from 1.8501 to 0.2412.

(2) The baselines that need no inverse bring limited gains. The format-invariance prompt barely changes FCC (88.46% → 88.64%), so instructing a model to ignore the layout does not remove its sensitivity. LLM normalization raises FCC only to 90.54%, needs one extra generation per structured record, and changes a number, unit, or negation term in 23.69% of the rewritten records.

These results confirm that canonicalization does not extend to unseen layouts and that LLM rewriting risks altering clinical details, whereas FORMEDER reduces format sensitivity on both seen and unseen formats with one forward pass per input. We have added the results of these baselines to the revised manuscript.

M2

"Cross-format consensus does not guarantee clinical correctness. ... It does not guarantee majority preference preservation."

A: Thank you for these insightful points! As you point out, the consensus is a cross-format reference rather than a clinical gold label, so errors shared by all views remain in it (Sec. 4). We therefore use it only to remove presentation-induced variation, and doing so does not lower correctness, as FORMEDER improves RA on all eight backbones (Tab. 1 of the paper).

It is true that equal averaging does not guarantee the majority direction, but we chose the arithmetic mean for two reasons. First, it is the unique distribution that minimizes ∑vDKL(pv∥q)\sum_v D_{\mathrm{KL}}(p_v\Vert q), and its preference score lies between the smallest and largest scores of the views (Prop. D.1), so a preference shared by all four views is always kept. Second, such cases are rare in practice. Among test neighborhoods with a 3–1 split, the consensus opposes the majority in only 1.32%–2.61% across the eight backbones, and a one-hot majority-vote target, which keeps the majority by construction, is worse on all three metrics (MedGemma-4B, Tab. 2 of the paper). In light of your example, we have revised the statement in Sec. 4 about deviations in a single view.

Following your suggestion, we further replace the consensus target with pooled conditional A/B probabilities or mean A/B log odds, with all else unchanged. The results are as follows.

Table 2. Consensus targets averaged over eight backbones.

Consensus targetFPS↓FCC↑ (%)RA↑ (%)
Arithmetic mean (FORMEDER)0.062296.2649.79
Conditional normalization0.064296.0850.31
Mean log odds0.065896.1251.21

The three targets perform similarly. The arithmetic mean gives the lowest FPS and highest FCC, while mean log odds gives higher RA, reflecting a trade-off between consistency and accuracy.

M3

"The combined evidence has no established correct diagnosis. Consequently, I think the experiment cannot measure correct conflict resolution. It can measure preference changes under format interventions."

A: Thank you for this thoughtful comment! As you note, without a joint gold diagnosis, our experiments measure preference changes under format interventions rather than correct conflict resolution. This matches our original goal of measuring and reducing how much the model's diagnostic preference depends on the format of the same evidence. Following your comment, we further examined whether reducing format sensitivity also helps the model resolve conflicts correctly, using two measures that need no gold diagnosis for the conflict. The first is the flip rate, the fraction of comparisons in which the chosen diagnosis changes with format. Each comparison keeps the evidence identical, so the correct answer is the same in every format, and a flip means the model is wrong in at least one format. The second is accuracy on single-record questions, whose correct answers are known. The results are as follows:

Table 3. Flip rate on conflicts (1 − FCC, seen formats, Tab. 11 of the paper) and single-record accuracy, averaged over eight backbones.

ModelFlip rate↓ (%)Mean accuracy over five formats↑ (%)RA↑ (%)
Frozen11.5454.3445.37
FORMEDER3.7456.9849.79

FORMEDER cuts the flip rate by about two-thirds, leaving far fewer comparisons that must contain an error. It also raises single-record accuracy, with RA gains on all eight backbones (Tab. 1 of the paper). Both results indicate that training toward the cross-format consensus leads the model to rely on the evidence rather than its layout, which correct conflict resolution requires.

We agree that presenting two patients' records as one patient is an artificial setting. We adopt it so that the records compete for a single answer and format sensitivity can be measured at scale. A case report usually documents one confirmed diagnosis and rarely contains two records of one patient that support different diagnoses. We therefore pair records from two patients matched on organ system and age (App. B), which also balances patients and diagnoses within each source pair.

We have accordingly clarified the scope of our claims. Observation 1 and the abstract now refer to PMC-ConForm, and the Limitations state that the conflict results measure format sensitivity, a prerequisite for correct conflict resolution, rather than the resolution itself.

Thank you also for the point on evidence informativeness! A preference for pathology may indeed reflect more specific evidence rather than a source bias, and we now note this in the Limitations. We would also like to clarify that this does not affect our main finding, namely that the same evidence leads to different preferences when only its format changes. The content of the records, and thus their specificity, is identical across formats, so it cannot explain these changes.

M4

"The authors show that averaging more presentations reduces disagreement. But isn't this behavior already mathematically expected? ... Does this experiment really justify the proposed consensus target?"

A: Thank you for pointing this out! We agree that the two subsets in the original Fig. 4 overlapped more as they grew, so part of the decline was expected by construction. Following your suggestion, we compared consensus predictions between disjoint groups of formats. On the 1,310 conflicts in the analysis set, we split the four structured formats into two disjoint groups and pooled one to six presentation states within each group.

Although the two groups share no structured format, the gap between their consensus preferences still decreases on all eight backbones as more states are pooled. This shows that the consensus becomes more stable with more presentations, which supports our consensus target. We have updated Fig. 4 and Sec. 3 accordingly.

M5

"Only 13 of 25 possible combinations are evaluated. Why exclude the remaining 12 combinations? ... Real medical records often contain mixed formats. Would the method still work in these settings?"

A: Thank you for raising this question! We originally selected these 13 states so that each preference shift could be attributed to a single format. Our evidence comes from narrative case reports, so we use prose as the reference format (Sec. 2). Each of the 13 states uses at most one structured format, alone or with prose. The other 12 states mix two structured formats, such as JSON and a table, so a shift there would depend on both at once.

As you point out, mixed formats are common in real records, so this setting is worth testing. Following your suggestion, we evaluate the checkpoints from Tab. 1 of the paper on all 72 format edges that involve at least one of the 12 mixed states. These checkpoints never see mixed states in training.

Table 4. Results on edges involving mixed states, averaged over eight backbones (RA uses single-record questions and is unaffected).

ModelFPS↓FCC↑ (%)
Frozen1.603788.44
FORMEDER0.062696.21

FORMEDER remains robust on mixed formats, with about 26× lower FPS than the frozen model. The same holds for the JSON–table combination you mention (FPS 1.5931 → 0.0627, FCC 88.46% → 96.18%).

M6

"The other baselines do not directly optimize cross-format consistency. But isn't consistency the main objective of ForMedER? Why not compare against symmetric KL or logit pairing?"

A: Thank you for your valuable suggestion! Methods that directly optimize cross-format consistency are indeed an important comparison, since consistency is central to FORMEDER. Following your suggestion, we further add two such baselines, symmetric KL and logit pairing. Each is combined with a distillation term toward the teacher's predictions and uses the same teacher, training data, and budget as FORMEDER. The results are as follows.

Table 5. Consistency-training baselines averaged over eight backbones.

MethodFPS↓FCC↑ (%)RA↑ (%)
Frozen1.607788.4645.37
Symmetric KL0.188694.9749.85
Logit pairing0.217894.7849.96
FORMEDER0.062296.2649.79

FORMEDER reduces format sensitivity more effectively than these consistency losses. It achieves 3.0×–3.5× lower FPS and higher FCC than both baselines, while RA remains comparable. We have added these results to the revised manuscript.

In addition, as you note, with a teacher-initialized student, the KL loss and gradient of Per-rendering KD are zero, so it reproduces the frozen model by construction (App. E.1). We have therefore removed it from Tab. 1 of the paper.

M7

"The main results also lack confidence intervals. Some RA improvements are relatively small. Are these improvements statistically significant?"

A: Thank you for this helpful suggestion! We agree that small gains need confidence intervals to show that they are reliable. We therefore compute paired 95% bootstrap confidence intervals for the RA gain of FORMEDER over the frozen model, resampling the 464 patient components of the test set (App. B.2).

Table 6. RA↑ gain of FORMEDER over the frozen model.

ModelRA↑ gain (95% CI)
MedGemma-4B3.20 (1.90, 4.50)
Gemma3-4B2.40 (1.23, 3.57)
HuatuoGPT-o1-7B2.70 (1.42, 3.94)
Qwen2.5-7B3.91 (2.69, 5.16)
BioMistral-7B18.72 (16.72, 20.76)
Mistral-7B1.12 (0.04, 2.20)
Llama3-8B2.14 (1.27, 3.05)
Qwen2.5-14B1.21 (0.36, 2.10)

All eight intervals exclude zero, indicating that the RA gains are significant on every backbone, including the smallest ones on Mistral-7B (1.12) and Qwen2.5-14B (1.21). We have added these intervals to the revised manuscript.

We hope our responses clarify the concerns raised. We have revised the manuscript accordingly. Thank you again for your careful review!

I thank Reviewer xyf4 for identifying a serious issue with how the results in Tables 9 and 10 were produced. The authors acknowledge that their records only partly explain how those values were obtained. Although they report that the rankings remain unchanged, I have lost confidence in the provenance and reproducibility of the affected entries. I am lowering my score from 3 (weak acceptance) to 1 (clear rejection) and flagging the submission for ethics review.

Reviewer_VZzhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the impact of evidence presentation formats on medical evidence-source preferences in large language models. The authors construct PMC-ConForm, a benchmark containing conflicting medical evidence presented in different formats, and find that format changes can significantly alter the models' diagnostic preferences. To address this issue, the authors propose ForMedER, a self-distillation framework that combines cross-format consensus learning and residual contraction. Experiments on eight large language models demonstrate that this method significantly improves format robustness and prediction consistency and can generalize to formats unseen during training.

Critical strengths

This paper focuses on biases caused by presentation formats in medical evidence reasoning, which has great practical significance. The authors propose the ForMedE framework to significantly reduce preference shifts caused by format changes.

Critical weaknesses

1.I have a question: the paper mainly evaluates format consistency, but does improving format consistency lead to improvements in the accuracy and reliability of medical reasoning?

2.The targets of this method are primarily derived from the predictions of a frozen teacher model. If the teacher model produces incorrect predictions, could the framework reinforce these incorrect judgments?

3.The current dataset mainly relies on artificially constructed conflicting evidence and predefined format templates, lacking experiments on real-world medical records.

Questions for authors

1.Does format correctness have a positive effect on reasoning correctness?

2.Does the ForMedER framework have an error-correction mechanism?

3.Can the authors validate the effectiveness of the method on real-world clinical records rather than artificially constructed conflicting evidence?

2 comments

Dear Reviewer VZzh, we thank you for the constructive comments and address each of your concerns below.

Weakness 1 & Question 1

"Does improving format consistency lead to improvements in the accuracy and reliability of medical reasoning?"

A: Good question! FORMEDER is designed to make the model focus on the content of the evidence during training and to reduce interference from its format, which may also improve its medical reasoning ability. To verify this, we measure accuracy on single-record questions, whose correct answer is the diagnosis of the patient who supplied the record, as conflict questions have no gold diagnosis. RA in Tab. 1 of the paper is computed on these questions, and FORMEDER improves it on all eight backbones (Sec. 5.1). However, RA requires a correct answer in all five formats and thus reflects consistency as well as accuracy. To check whether FORMEDER improves accuracy itself, we further report prose accuracy and mean accuracy over the five formats, which measure accuracy alone:

Table 1. Single-record accuracy averaged over eight backbones.

ModelProse accuracy↑ (%)Mean accuracy over five formats↑ (%)RA↑ (%)
Frozen55.4254.3445.37
FORMEDER56.2956.9849.79

FORMEDER raises mean accuracy over the five formats as well as RA, showing that both consistency and accuracy improve.

Weakness 2 & Question 2

"If the teacher model produces incorrect predictions, could the framework reinforce these incorrect judgments?"

A: Thank you for this insightful question! As you point out, FORMEDER's target comes from the frozen model's own predictions, namely their cross-format consensus (Sec. 4), so an error made in every format also appears in the target. However, FORMEDER can correct errors that appear in only some formats. Such an error arises from the presentation rather than from missing knowledge, since the model reaches the correct diagnosis from the same record in another format. By reducing interference from formats, FORMEDER makes these presentation-induced errors less likely. To test whether training corrects these errors or spreads them to all formats, we use the single-record questions from W1, whose correct answers are known, and count how many of the five formats each evaluation unit answers correctly before and after training.

Table 2. Single-record evaluation units by the number of formats answered correctly, summed over eight backbones.

FrozenFORMEDER: 5 correctFORMEDER: 1–4 correctFORMEDER: 0 correct
5 correct18,375553265
1–4 correct1,6805,115647
0 correct1,01066113,998

As shown, FORMEDER reduces rather than reinforces errors. Units correct in all five formats increase from 19,193 to 21,065, and units wrong in all five decrease from 15,669 to 14,910. Among units with errors in only some formats, 1,680 become fully correct and only 647 become fully wrong, so FORMEDER corrects such errors far more often than it spreads them.

Weakness 3 & Question 3

"The current dataset mainly relies on artificially constructed conflicting evidence and predefined format templates, lacking experiments on real-world medical records."

A: Thank you for this valuable suggestion! We fully agree that validation on real clinical records would further strengthen our conclusions. We would like to clarify that PMC-ConForm is built on real clinical data. Each record comes from a real patient's case report in PMC-Patients (App. B), and only the pairing of two patients' records into a conflict is constructed. However, direct validation on real clinical records is difficult for three reasons.

(1) Real conflicts are hard to obtain and unlabeled. Cases in which two records point to different diagnoses are hard to collect at scale, and each would need clinicians to confirm the conflict, label the sources, and determine the diagnosis, with enough cases to cover every pair of sources.

(2) Privacy rules restrict the use and release of clinical records. Privacy regulations and data-use agreements limit how clinical records can be shared and may prevent us from releasing the data for reproduction.

(3) Our metrics need the same evidence in several layouts, which clinical records rarely provide. A clinical record usually exists in only one layout, and two documents in the same chart differ in both content and layout, so they cannot isolate the effect of format. Re-rendering real records would again require templates.

Nevertheless, FORMEDER still lowers FPS on four layouts unseen in training (Sec. 5.2). Real clinical records also come in layouts not seen in training, so this result offers initial support for applying FORMEDER to them. We leave validation on real clinical records to future work.

We hope our responses clarify the concerns raised. Thank you again for your insightful comments!

Thank you to the authors for their response, which addressed most of my previous concerns. However, I still have one major concern: the medical evidence conflicts in this paper are artificially constructed by combining records from two different patients, which is quite different from conflicts involving the same patient in real clinical settings. I believe resolving conflicts between different medical records of the same patient would be more valuable and challenging. Nevertheless, I still appreciate the contributions of this work and have decided to maintain my current positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.