- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces ContextAdapt, an evaluation framework for testing whether LLMs apply abstract alignment values differently when professional context changes the applicable norm, while remaining stable when contextual variation should not change the norm. The benchmark covers honesty, autonomy, and confidentiality across medicine, law, finance, and national security. The authors construct a professional-norm corpus from 24 primary-source professional, regulatory, governmental, and institutional documents, derive a value-by-domain matrix, and build two scenario datasets: a main dataset of 107 scenarios testing default rules and exceptions, and a secondary factorial dataset of 144 scenarios varying stakes, agent role, and explicit/implicit domain labels. The paper evaluates 12 LLMs with 10 samples per model/scenario and scores open-ended responses using a three-LLM judge panel on both action appropriateness and stated justification. The main finding is that models have high mean behavioural appropriateness, but vary substantially in whether they provide the correct domain-specific justification; in the secondary experiment, stakes sometimes cause large behavioural shifts even when the underlying professional obligation is intended to remain unchanged.
Critical strengths
-
The paper identifies an important and under-tested alignment capability. The distinction between following an abstract value and operationalising that value under a profession-specific norm is conceptually strong. The paper usefully argues that alignment evaluations should test both adaptation when norms change and invariance when superficial context changes do not alter the governing norm.
-
The benchmark design is thoughtful and more grounded than generic moral-preference benchmarks. Building scenarios from a corpus of professional and regulatory sources, separating action appropriateness from stated justification, and including a secondary factorial experiment on stakes/role/domain-label manipulations are all consequential design choices. They make the benchmark more diagnostic than a simple value-agreement test.
Critical weaknesses
-
The scenario space is too coarse for the breadth of the claim. The experiment is not small in terms of generated model responses: it covers many model outputs, 12 models, repeated sampling, and a three-judge evaluation pipeline. However, the underlying scenario coverage is much narrower. The main dataset contains 107 scenarios across three values and four professional domains, which averages to fewer than 10 scenarios per value-domain cell before considering default rules, exceptions, roles, jurisdictions, and subtypes. The secondary factorial dataset has 144 scenarios, but these appear to come from a much smaller set of base templates varied by stakes, role, and explicit/implicit domain label. This is a concern because medicine, law, finance, and national security each contain many subdomains where honesty, autonomy, and confidentiality are operationalised differently. For example, medical disclosure norms differ across emergency care, oncology, psychiatry, pediatrics, surrogate decision-making, public-health reporting, and end-of-life care; legal confidentiality differs across criminal defense, corporate representation, fraud exceptions, conflicts of interest, and duties to the court; financial obligations differ across retail advice, market disclosure, AML reporting, audit, and fiduciary duty. The benchmark is a useful first testbed, but the current scenario taxonomy is too coarse to support broad claims about robust context-adaptive value alignment across professional domains.
-
The action-level task may be too easy or too coarsely labeled, so the headline appropriateness score is hard to interpret. The paper reports 95.6% mean behavioural appropriateness on the main dataset, with high scores across most domain-value combinations. This raises the possibility that the reference actions are often broad, conservative, or professionally plausible enough that models can select the right action without genuinely applying the intended professional norm. The justification results are more revealing: correct domain-specific justification ranges from 25.6% to 76.9% across models, suggesting that many models arrive at appropriate actions without grounding them in the relevant professional framework. This interpretation is further complicated by the reliance on LLM judges and by the lack of reported validation from domain experts. Because the paper's central claim depends on fine distinctions in professional norms, it would be important to know whether doctors, lawyers, financial compliance professionals, or national-security experts agree with the scenario design, reference outcomes, exception boundaries, and justification labels.
Questions for authors
-
Can you provide per-cell scenario counts for each value × domain combination, separated by default-rule and exception cases? This would help readers judge how much of each professional value space is actually covered.
-
How do you distinguish genuine context-adaptive value reasoning from models selecting broadly safe or plausible professional actions? Given the 95.6% action-level appropriateness score, please discuss whether the behavioural task is sufficiently discriminative.
-
Did any qualified professionals or legal/medical/financial/national-security domain experts validate the value-domain matrix, reference outcomes, exception boundaries, and justification categories? If so, please report the process and agreement; if not, please make that limitation more central.
-
How sensitive are the main conclusions to the LLM judge panel? Given the moderate/low agreement for outcome and justification labels, do the headline results hold under each judge separately, or under a human-audited subset?
4 comments
Context-adaptive reasoning and professional validation
2. Does 95.9% action appropriateness demonstrate genuine context-adaptive reasoning?
The aggregate Main score establishes that models often select actions corresponding to the benchmark reference; it does not establish that they internally applied the intended professional rule, nor does it rule out broadly safe or professionally plausible policies.
This distinction is why ContextAdapt reports action and stated professional grounding separately:
| Main-dataset measure | Result |
|---|---|
| Mean action appropriateness | 95.9% |
| Correct professional-framework justification, pooled | 46.1% |
| Correct-framework range across models | 25.6–76.9% |
| Domain-neutral constitutional justification, pooled | 42.1% |
| Generic moral heuristic, pooled | 10.7% |
The substantial gap between action-level appropriateness and explicit professional grounding shows why the 95.9% score should not be interpreted as evidence of “reasoning” in the stronger sense.
We therefore frame the two datasets as complementary behavioural tests rather than as direct access to internal reasoning. This has been clarified in the text of the manuscript. Main Exception probes test documented departures from default professional rules. The Secondary experiment tests invariance: matched variants manipulate role, explicit domain naming, or stakes while retaining the same benchmark reference within each pair. Stakes variants are specifically constructed to remain on the same side of the relevant documented professional boundary.
The pooled results are:
| Secondary manipulation | Condition 1 | Condition 2 | Gap |
|---|---|---|---|
| Domain label | Explicit 89.1% | Implicit 88.1% | +1.0 pt |
| Agent role | Decision-maker 89.0% | Advisor 88.1% | +0.9 pt |
| Stakes | High 93.7% | Low 83.4% | +10.3 pts (High − Low) |
Thus role and explicit naming have little aggregate effect, while stakes produce much larger behavioural changes despite the matched variants sharing the same benchmark reference. We have also become more cautious about the Exception result: the aggregate Exception-Base difference is -2.5 points and ranges from -5.7 to +1.2 points when individual Main templates are removed. We therefore do not present this small aggregate difference as a general benchmark-level result.
In short, we agree with the reviewer’s central point: high action accuracy alone is insufficient to distinguish contextual adaptation from a broadly plausible policy. Our revised claims explicitly separate action appropriateness, stated professional grounding, adaptation to documented exception conditions, and invariance under contextual changes that should not alter the reference. A larger set of tightly matched cases in which the appropriate action itself flips would be a useful extension, but we do not claim to have established internal value reasoning from the present action score alone.
3. Professional validation
No qualified medical, legal, financial, or national-security professionals independently validated the complete Value × Domain matrix, reference outcomes, exception boundaries, or justification taxonomy. We agree this is an important limitation and state it explicitly in the revised paper. ContextAdapt should be interpreted as source-grounded rather than professionally certified ground truth.
The normative corpus was constructed from 45 candidate instruments identified through major international and European professional and regulatory bodies, together with major bodies in the represented jurisdictions, particularly the UK and US, and narrowed to 24 retained sources:
| Domain | Candidates | Retained |
|---|---|---|
| Medicine | 9 | 9 |
| Law | 8 | 5 |
| Finance | 14 | 7 |
| National Security | 14 | 3 |
| Total | 45 | 24 |
Relevant provisions were extracted by the first author and the resulting operationalisations checked by the co-author. This improves transparency and author-level checking but does not substitute for independent domain-expert validation, which remains important future work.
Sensitivity to the LLM judge panel
4. Sensitivity to the LLM judge panel
We recomputed the headline statistics using each judge independently:
| Rater | Main appropriateness | Correct-framework share (model range) | High − Low stakes | Role gap | Label gap |
|---|---|---|---|---|---|
| Panel consensus | 95.9% | 46.1% (25.6–76.9%) | +10.3 | +0.9 | +1.0 |
| GPT-5.6 Luna | 93.1% | 50.6% (29.1–78.7%) | +9.2 | +1.3 | +0.6 |
| Gemini 3.5 Flash-Lite | 95.4% | 49.2% (32.3–79.1%) | +11.3 | +1.2 | +1.2 |
| Command A | 96.8% | 28.9% (8.5–59.4%) | +9.5 | +0.7 | +0.3 |
Under every individual judge:
- Main action appropriateness remains near ceiling (93.1–96.8%);
- the High−Low stakes effect remains approximately 9–11 points;
- role and label effects remain at most 1.3 points; and
- the cross-model ordering of professional grounding remains highly stable.
Pairwise Spearman correlations between individual judges for the correct-framework model ranking are 0.89-0.98, and every judge identifies Gemma 2 27B as the lowest and Gemini 3.5 Flash-Lite as the highest on this measure.
We agree that absolute labels are less certain. Pairwise judge agreement is only moderate: outcome Cohen’s κ is approximately 0.34-0.58, while justification Krippendorff’s α is approximately 0.22-0.51. Correspondingly, absolute correct-framework rates differ substantially between Command A and the other judges. We therefore interpret absolute justification percentages more cautiously than the stable cross-model ordering.
Some individual cell-level effects are less stable across judges than these aggregate findings. For example:
| Rater | NS institution-facing confidentiality: High | Low | High − Low |
|---|---|---|---|
| Panel consensus | 88.2% | 9.9% | +78.3 |
| GPT-5.6 Luna | 75.0% | 75.6% | −0.6 |
| Gemini 3.5 Flash-Lite | 83.4% | 9.5% | +73.9 |
| Command A | 98.5% | 2.7% | +95.8 |
Thus, the very large panel-consensus National Security example is reproduced by Gemini and Command A but not GPT-5.6 Luna. We therefore treat this cell as illustrative and judge-sensitive rather than as a benchmark-level conclusion.
Finally, the conclusions are also robust to benchmark composition and averaging:
| Analysis | Main appropriateness | Correct framework | High − Low stakes |
|---|---|---|---|
| Full pooled analysis | 95.9% | 46.1% | +10.3 |
| Macro-average | 96.2% | 46.9% | +11.6 |
| Without National Security | 96.4% | 53.7% | +7.6 |
| Without National Security, macro | 96.6% | 53.9% | +8.0 |
Taken together, these analyses support that the action-level benchmark is near ceiling, but professional grounding remains substantially more variable, and the principal stakes, role, domain-label, and professional-grounding findings are robust across judges, domains, scenario families, and averaging schemes.
Thank you for the thoughtful rebuttal. I appreciate the additional judge-sensitivity analyses and the clearer distinction between action appropriateness and professional grounding. The revisions also make the limitations around scenario coverage and independent expert validation clearer. I am keeping my original rating of 3 (Weak Accept).
Scenario coverage
Thank you for the thoughtful review. We have revised the paper to address scenario coverage, interpretation of the high action-level score, professional validation, and sensitivity to the LLM judge panel.
Note on corrected results: During rebuttal we identified output truncation in affected model runs, regenerated those responses with corrected output limits, and re-judged them. The corrected Main mean is 95.9%, compared with 95.6% in the submitted manuscript; pooled headline estimates change only slightly (by at most 0.3 percentage points). All results below use the corrected outputs.
1. Scenario coverage
The 107 Main scenarios comprise 10 higher-level templates and 39 distinct sub-templates: 80 Base, 24 Exception, and 3 Edge scenarios. The complete per-cell breakdown is:
We agree that this does not exhaust the many subdomains of medicine, law, finance, or national security. We have therefore narrowed language that could imply comprehensive professional coverage and describe ContextAdapt as a controlled evaluation of the normative structures represented in the selected source corpus.
We considered adding a small number of new subdomain cases during rebuttal, but without corresponding expert validation, which was not possible in the limited time available, this would not address the reviewer’s broader concern about professional coverage. Instead, we tested dependence on the existing scenario composition. Main appropriateness remains 95.2–96.6% after removing any single Main template. In the Secondary experiment, removing any one of the 18 underlying scenario families leaves the High−Low stakes gap between +7.1 and +12.3 points, while role and explicit/implicit-label effects remain at most 1.4 points.