acceptodds
Under review as a conference paper at ICLR 2027

ContextAdapt: Evaluating Contextual Adaptation and Value Alignment in LLMs

Abstract

Values such as honesty, autonomy, and confidentiality are often regarded as general principles underpinning AI alignment. However, what it means to act in accordance with these values can depend on the context in which a decision is made. In this paper, we ask whether large language models (LLMs) appropriately adapt the application of a value across professional settings, while remaining consistent when contextual changes do not alter the relevant professional norm. To study this, we introduce ContextAdapt, an evaluation framework covering honesty, autonomy, and confidentiality across medicine, law, finance, and national security. Drawing on primary-source professional and regulatory documents, we construct a value x domain framework and use this to develop scenarios testing both default professional rules and recognised exceptions. We evaluate 12 LLMs on both the actions they recommend and the justifications they provide. In our main experiment, models achieve 95.6% mean appropriateness, although the use of the correct domain-specific justification varies substantially across models, from 25.6% to 76.9%. In a separate factorial experiment, explicitly naming the professional domain and changing the role of the model have limited effect on behaviour. Varying stakes, however, reveals severe but localised failures: in some cases, models alter their responses even though the underlying professional obligation remains unchanged. In particular, perceived severity appears to act as a cue for disclosure across both honesty and confidentiality scenarios. These results show that evaluating value alignment requires us to consider not only whether models follow abstract principles, but whether they apply them appropriately across different contexts.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI alignment, large language models, value alignment, contextual reasoning, LLM evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3uPxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ContextAdapt, an evaluation framework for testing whether LLMs apply abstract alignment values differently when professional context changes the applicable norm, while remaining stable when contextual variation should not change the norm. The benchmark covers honesty, autonomy, and confidentiality across medicine, law, finance, and national security. The authors construct a professional-norm corpus from 24 primary-source professional, regulatory, governmental, and institutional documents, derive a value-by-domain matrix, and build two scenario datasets: a main dataset of 107 scenarios testing default rules and exceptions, and a secondary factorial dataset of 144 scenarios varying stakes, agent role, and explicit/implicit domain labels. The paper evaluates 12 LLMs with 10 samples per model/scenario and scores open-ended responses using a three-LLM judge panel on both action appropriateness and stated justification. The main finding is that models have high mean behavioural appropriateness, but vary substantially in whether they provide the correct domain-specific justification; in the secondary experiment, stakes sometimes cause large behavioural shifts even when the underlying professional obligation is intended to remain unchanged.

Critical strengths
  1. The paper identifies an important and under-tested alignment capability. The distinction between following an abstract value and operationalising that value under a profession-specific norm is conceptually strong. The paper usefully argues that alignment evaluations should test both adaptation when norms change and invariance when superficial context changes do not alter the governing norm.

  2. The benchmark design is thoughtful and more grounded than generic moral-preference benchmarks. Building scenarios from a corpus of professional and regulatory sources, separating action appropriateness from stated justification, and including a secondary factorial experiment on stakes/role/domain-label manipulations are all consequential design choices. They make the benchmark more diagnostic than a simple value-agreement test.

Critical weaknesses
  1. The scenario space is too coarse for the breadth of the claim. The experiment is not small in terms of generated model responses: it covers many model outputs, 12 models, repeated sampling, and a three-judge evaluation pipeline. However, the underlying scenario coverage is much narrower. The main dataset contains 107 scenarios across three values and four professional domains, which averages to fewer than 10 scenarios per value-domain cell before considering default rules, exceptions, roles, jurisdictions, and subtypes. The secondary factorial dataset has 144 scenarios, but these appear to come from a much smaller set of base templates varied by stakes, role, and explicit/implicit domain label. This is a concern because medicine, law, finance, and national security each contain many subdomains where honesty, autonomy, and confidentiality are operationalised differently. For example, medical disclosure norms differ across emergency care, oncology, psychiatry, pediatrics, surrogate decision-making, public-health reporting, and end-of-life care; legal confidentiality differs across criminal defense, corporate representation, fraud exceptions, conflicts of interest, and duties to the court; financial obligations differ across retail advice, market disclosure, AML reporting, audit, and fiduciary duty. The benchmark is a useful first testbed, but the current scenario taxonomy is too coarse to support broad claims about robust context-adaptive value alignment across professional domains.

  2. The action-level task may be too easy or too coarsely labeled, so the headline appropriateness score is hard to interpret. The paper reports 95.6% mean behavioural appropriateness on the main dataset, with high scores across most domain-value combinations. This raises the possibility that the reference actions are often broad, conservative, or professionally plausible enough that models can select the right action without genuinely applying the intended professional norm. The justification results are more revealing: correct domain-specific justification ranges from 25.6% to 76.9% across models, suggesting that many models arrive at appropriate actions without grounding them in the relevant professional framework. This interpretation is further complicated by the reliance on LLM judges and by the lack of reported validation from domain experts. Because the paper's central claim depends on fine distinctions in professional norms, it would be important to know whether doctors, lawyers, financial compliance professionals, or national-security experts agree with the scenario design, reference outcomes, exception boundaries, and justification labels.

Questions for authors
  1. Can you provide per-cell scenario counts for each value × domain combination, separated by default-rule and exception cases? This would help readers judge how much of each professional value space is actually covered.

  2. How do you distinguish genuine context-adaptive value reasoning from models selecting broadly safe or plausible professional actions? Given the 95.6% action-level appropriateness score, please discuss whether the behavioural task is sufficiently discriminative.

  3. Did any qualified professionals or legal/medical/financial/national-security domain experts validate the value-domain matrix, reference outcomes, exception boundaries, and justification categories? If so, please report the process and agreement; if not, please make that limitation more central.

  4. How sensitive are the main conclusions to the LLM judge panel? Given the moderate/low agreement for outcome and justification labels, do the headline results hold under each judge separately, or under a human-audited subset?

4 comments
AuthorsOpenReview ↗

Scenario coverage

Thank you for the thoughtful review. We have revised the paper to address scenario coverage, interpretation of the high action-level score, professional validation, and sensitivity to the LLM judge panel.

Note on corrected results: During rebuttal we identified output truncation in affected model runs, regenerated those responses with corrected output limits, and re-judged them. The corrected Main mean is 95.9%, compared with 95.6% in the submitted manuscript; pooled headline estimates change only slightly (by at most 0.3 percentage points). All results below use the corrected outputs.

1. Scenario coverage

The 107 Main scenarios comprise 10 higher-level templates and 39 distinct sub-templates: 80 Base, 24 Exception, and 3 Edge scenarios. The complete per-cell breakdown is:

DomainHonestyAutonomyConfidentialityTotal
Medicine8 (5 Base, 3 Exception)11 (5 Base, 3 Exception, 3 Edge)8 (5 Base, 3 Exception)27
Law5 Base12 (9 Base, 3 Exception)8 (5 Base, 3 Exception)25
Finance5 Base12 (9 Base, 3 Exception)8 (5 Base, 3 Exception)25
National Security9 Base12 (9 Base, 3 Exception)9 Base30
Total274733107

We agree that this does not exhaust the many subdomains of medicine, law, finance, or national security. We have therefore narrowed language that could imply comprehensive professional coverage and describe ContextAdapt as a controlled evaluation of the normative structures represented in the selected source corpus.

We considered adding a small number of new subdomain cases during rebuttal, but without corresponding expert validation, which was not possible in the limited time available, this would not address the reviewer’s broader concern about professional coverage. Instead, we tested dependence on the existing scenario composition. Main appropriateness remains 95.2–96.6% after removing any single Main template. In the Secondary experiment, removing any one of the 18 underlying scenario families leaves the High−Low stakes gap between +7.1 and +12.3 points, while role and explicit/implicit-label effects remain at most 1.4 points.

AuthorsOpenReview ↗

Context-adaptive reasoning and professional validation

2. Does 95.9% action appropriateness demonstrate genuine context-adaptive reasoning?

The aggregate Main score establishes that models often select actions corresponding to the benchmark reference; it does not establish that they internally applied the intended professional rule, nor does it rule out broadly safe or professionally plausible policies.

This distinction is why ContextAdapt reports action and stated professional grounding separately:

Main-dataset measureResult
Mean action appropriateness95.9%
Correct professional-framework justification, pooled46.1%
Correct-framework range across models25.6–76.9%
Domain-neutral constitutional justification, pooled42.1%
Generic moral heuristic, pooled10.7%

The substantial gap between action-level appropriateness and explicit professional grounding shows why the 95.9% score should not be interpreted as evidence of “reasoning” in the stronger sense.

We therefore frame the two datasets as complementary behavioural tests rather than as direct access to internal reasoning. This has been clarified in the text of the manuscript. Main Exception probes test documented departures from default professional rules. The Secondary experiment tests invariance: matched variants manipulate role, explicit domain naming, or stakes while retaining the same benchmark reference within each pair. Stakes variants are specifically constructed to remain on the same side of the relevant documented professional boundary.

The pooled results are:

Secondary manipulationCondition 1Condition 2Gap
Domain labelExplicit 89.1%Implicit 88.1%+1.0 pt
Agent roleDecision-maker 89.0%Advisor 88.1%+0.9 pt
StakesHigh 93.7%Low 83.4%+10.3 pts (High − Low)

Thus role and explicit naming have little aggregate effect, while stakes produce much larger behavioural changes despite the matched variants sharing the same benchmark reference. We have also become more cautious about the Exception result: the aggregate Exception-Base difference is -2.5 points and ranges from -5.7 to +1.2 points when individual Main templates are removed. We therefore do not present this small aggregate difference as a general benchmark-level result.

In short, we agree with the reviewer’s central point: high action accuracy alone is insufficient to distinguish contextual adaptation from a broadly plausible policy. Our revised claims explicitly separate action appropriateness, stated professional grounding, adaptation to documented exception conditions, and invariance under contextual changes that should not alter the reference. A larger set of tightly matched cases in which the appropriate action itself flips would be a useful extension, but we do not claim to have established internal value reasoning from the present action score alone.

3. Professional validation

No qualified medical, legal, financial, or national-security professionals independently validated the complete Value × Domain matrix, reference outcomes, exception boundaries, or justification taxonomy. We agree this is an important limitation and state it explicitly in the revised paper. ContextAdapt should be interpreted as source-grounded rather than professionally certified ground truth.

The normative corpus was constructed from 45 candidate instruments identified through major international and European professional and regulatory bodies, together with major bodies in the represented jurisdictions, particularly the UK and US, and narrowed to 24 retained sources:

DomainCandidatesRetained
Medicine99
Law85
Finance147
National Security143
Total4524

Relevant provisions were extracted by the first author and the resulting operationalisations checked by the co-author. This improves transparency and author-level checking but does not substitute for independent domain-expert validation, which remains important future work.

AuthorsOpenReview ↗

Sensitivity to the LLM judge panel

4. Sensitivity to the LLM judge panel

We recomputed the headline statistics using each judge independently:

RaterMain appropriatenessCorrect-framework share (model range)High − Low stakesRole gapLabel gap
Panel consensus95.9%46.1% (25.6–76.9%)+10.3+0.9+1.0
GPT-5.6 Luna93.1%50.6% (29.1–78.7%)+9.2+1.3+0.6
Gemini 3.5 Flash-Lite95.4%49.2% (32.3–79.1%)+11.3+1.2+1.2
Command A96.8%28.9% (8.5–59.4%)+9.5+0.7+0.3

Under every individual judge:

  • Main action appropriateness remains near ceiling (93.1–96.8%);
  • the High−Low stakes effect remains approximately 9–11 points;
  • role and label effects remain at most 1.3 points; and
  • the cross-model ordering of professional grounding remains highly stable.

Pairwise Spearman correlations between individual judges for the correct-framework model ranking are 0.89-0.98, and every judge identifies Gemma 2 27B as the lowest and Gemini 3.5 Flash-Lite as the highest on this measure.

We agree that absolute labels are less certain. Pairwise judge agreement is only moderate: outcome Cohen’s κ is approximately 0.34-0.58, while justification Krippendorff’s α is approximately 0.22-0.51. Correspondingly, absolute correct-framework rates differ substantially between Command A and the other judges. We therefore interpret absolute justification percentages more cautiously than the stable cross-model ordering.

Some individual cell-level effects are less stable across judges than these aggregate findings. For example:

RaterNS institution-facing confidentiality: HighLowHigh − Low
Panel consensus88.2%9.9%+78.3
GPT-5.6 Luna75.0%75.6%−0.6
Gemini 3.5 Flash-Lite83.4%9.5%+73.9
Command A98.5%2.7%+95.8

Thus, the very large panel-consensus National Security example is reproduced by Gemini and Command A but not GPT-5.6 Luna. We therefore treat this cell as illustrative and judge-sensitive rather than as a benchmark-level conclusion.

Finally, the conclusions are also robust to benchmark composition and averaging:

AnalysisMain appropriatenessCorrect frameworkHigh − Low stakes
Full pooled analysis95.9%46.1%+10.3
Macro-average96.2%46.9%+11.6
Without National Security96.4%53.7%+7.6
Without National Security, macro96.6%53.9%+8.0

Taken together, these analyses support that the action-level benchmark is near ceiling, but professional grounding remains substantially more variable, and the principal stakes, role, domain-label, and professional-grounding findings are robust across judges, domains, scenario families, and averaging schemes.

Thank you for the thoughtful rebuttal. I appreciate the additional judge-sensitivity analyses and the clearer distinction between action appropriateness and professional grounding. The revisions also make the limitations around scenario coverage and independent expert validation clearer. I am keeping my original rating of 3 (Weak Accept).

Reviewer_xZ6yOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a benchmark, ContextAdapt, to evaluate whether LLMs appropriately adapt the application of three abstract values, honestly, autonomy, and confidentiality, in four specific professional domains: medicine, law, finance, and national security.

The authors construct a corpus of 24 professional standard documents, derive context-dependent normative rules, and evaluate 12 LLMs using 107 main scenarios and 144 secondary scenarios.

Critical strengths
  1. The motivation of this paper is clear, it identifies a meaningful limitation of value-based alignment.
Critical weaknesses
  1. The paper is difficult to follow in several places, as important tables and supporting results are presented in the appendix rather than near the corresponding discussions in the main text. Moreover, Figure 2 includes six categories in its legend, but I can only identify four of them in the figure, making it challenging to interpret the results. The introduction lacks an explicit summary of the paper's main contributions, making it difficult for readers to identify the novelty and significance of the proposed benchmark relative to existing work.

  2. The author mentioned that they identify three features of the professional norm. It is unclear how the author identifies these three features and how to verify the effectiveness of these features. It is crucial for a benchmark paper.

Questions for authors
  1. The authors state that "Across models, the proportion of responses classified as using the correct domain-specific justification ranges from 25.6% to 76.9% (Figure 2)." However, these specific values are not explicitly annotated in Figure 2, making them difficult to verify from the visualization. Could the authors provide the corresponding numerical results, ideally in a table, and clarify which models achieve these minimum and maximum values?

  2. The 24 source documents are unevenly distributed in the four professional domains: seven for finance, five for law, nine for medicine, and three for national security. This imbalance raises concerns about whether the professional norms are represented with comparable depth and completeness in all domains. Could the authors explain the criteria used to select these documents and discuss whether the uneven sources might affect the reliability of the reference outcomes or the validity of cross-domain comparisons?

5 comments
AuthorsOpenReview ↗

Main contributions and updated figure

Thank you for the detailed comments. We have revised both the presentation and methodological description, and report the key information directly here so that the response does not depend on consulting the revised manuscript.

Note on corrected results: During rebuttal we identified output truncation in affected model runs, regenerated those responses with corrected output limits, and re-judged them. This changes the submitted Main-dataset mean from 95.6% to 95.9%; the pooled headline estimates change only slightly (by at most 0.3 percentage points). All results below use the corrected outputs.

1. Main contributions and placement of results

We added an explicit three-part contribution summary to the Introduction and moved key evidence closer to the claims it supports. In particular, the main text now reports the complete per-cell benchmark coverage, the pooled domain × value appropriateness results, the revised justification figure with numerical annotations, and the judge-sensitivity analysis.

The main findings are:

  • Main-dataset mean action appropriateness: 95.9%.
  • Appropriateness across the 12 domain × value cells: 90.6–99.4%.
  • Correct domain-specific justification: 25.6–76.9% across models, with a pooled value of 46.1%.
  • Secondary Explicit vs. Implicit domain label: 89.1% vs. 88.1%. -Decision-maker vs. Advisor: 89.0% vs. 88.1%.
  • High vs. Low stakes: 93.7% vs. 83.4%, a +10.3-point High−Low difference.

2. Figure 2 and numerical justification results

We agree that the original figure made the two low-frequency categories appear absent. We regenerated Figure 2 so that all six justification categories are numerically annotated for every model, including zeros. Explicit uncertainty and Domain-specific (wrong domain) are genuinely rare (each accounts for at most 0.2% of any model’s responses), which is why they were previously almost invisible.

For completeness, the exact values underlying the revised figure are:

ModelCorrect domain-specificDomain-neutralGeneric moralUncertaintyWrong domainNone
Gemini 3.5 Flash-Lite76.9%20.5%2.5%00.1%0
Command A+†63.0%30.1%6.8%000.1%
Gemma 3 27B61.9%33.4%4.7%000
Claude Sonnet 554.1%39.2%6.7%000
DeepSeek V4 Flash49.7%42.8%7.4%0.1%00.1%
GPT-5.6 Luna45.6%41.7%12.6%0.1%00
Claude Haiku 4.545.3%45.5%8.9%0.2%00
Llama 4 Scout44.2%43.7%11.9%0.2%00
Command A41.4%40.2%8.6%009.9%
Llama 3.3 70B40.5%46.6%12.9%000
Llama 3.1 70B36.2%46.7%16.9%0.1%00.1%
Llama 3.1 8B30.8%46.7%21.0%001.5%
Gemma 2 27B25.6%59.1%15.1%0.2%00
Pooled, 12 models46.1%42.1%10.7%0.1%<0.1%1.0%
†Command A+ was run on the Secondary experiment, but API rate limits left incomplete Secondary coverage. It is therefore excluded from all pooled statistics; its complete Main-dataset result is reported separately.
AuthorsOpenReview ↗

Professional norms and source coverage

3. How the professional norms were constructed

We agree that the submitted paper did not explain this sufficiently. The three features represented in the Value × Domain matrix are:

  • the default operationalisation of the value in that professional setting;
  • the principal target or beneficiary of the obligation; and
  • any documented exception or override together with its triggering conditions and procedural requirements.

These features correspond directly to benchmark construction: the default determines Base reference outcomes; the target distinguishes who or what the obligation is owed to, including person-facing versus institution-facing cases; and documented override conditions determine Exception probes.

We identified the initial source set by reviewing major international and European professional and regulatory bodies, together with major bodies in the jurisdictions represented in the study, particularly the UK and US, and locating relevant ethics codes, conduct standards, and closely related normative guidance. This yielded 45 candidate instruments:

DomainCandidatesRetained
Medicine99
Law85
Finance147
National Security143
Total4524

For each candidate we recorded the issuing body, jurisdiction, source type, and relevance to the focal values. Professional self-regulatory codes were prioritised. Where no suitable practitioner-facing code was available, or where an international or regulatory framework directly supplied a norm used in the matrix, we also retained relevant governmental, institutional, international, or regulatory frameworks. Sources were otherwise set aside when they governed a materially different role or did not directly add relevant normative content.

For the retained sources, relevant provisions were extracted under honesty, autonomy, and confidentiality by the first author, and the resulting operationalisations were checked by the co-author. This provides source grounding and author-level checking but does not substitute for independent domain-expert validation, which we now state prominently as a limitation.

4. Uneven source coverage

The corpus was not constructed to equalise document counts. Sources were selected for practitioner relevance and normative coverage; a single code can contain several relevant obligations, while multiple documents can substantially repeat the same norm.

We nevertheless agree that National Security’s smaller three-document corpus creates greater uncertainty. We therefore tested both the effect of excluding that domain and the effect of equal-weight averaging:

AnalysisMain appropriatenessCorrect frameworkHigh − Low stakes gap
Pooled analysis95.9%46.1%+10.3 pts
Equal-weight macro-average96.2%46.9%+11.6 pts
Excluding National Security96.4%53.7%+7.6 pts
Excluding NS, macro-average96.6%53.9%+8.0 pts

Thus, neither unequal scenario counts nor the smaller National Security corpus drives the central conclusions.

We thank the reviewer for identifying these presentation and methodological gaps. The revised version makes both the construction of the benchmark and the numerical evidence supporting its conclusions substantially easier to inspect.

I appreciate the authors' detailed responses, but I still have some concerns about your rebuttal. 1. Most of the contribution statement describes experimental procedures and conclusions rather than clearly establishing the novelty of the work. The authors have not clearly claimed what knowledge or methodological innovation these experiments provide beyond reporting benchmark results. 2. Although the revised Figure 2 improves the readability of the justification-type distribution, there remains an inconsistency between the scope of the visualization and its corresponding statistical analysis. Figure 2 presents results for 13 models on the main dataset, whereas the accompanying discussion reports pooled statistics based on only 12 models, excluding Command A+. Given that Command A+ has complete coverage of the main dataset, it is unclear why its incomplete coverage of the secondary experiment should affect the statistical analysis of the main dataset. This inconsistency may confuse readers who naturally expect the reported pooled statistics to correspond to the models visualized in Figure 2.

Reviewer_KejyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors focused their work on investigating how well language models the performance of language models in professional settings. They created an evaluation framework called CONTEXTADAPT to allow them examine how well language models change and operationalize values such as honesty, autonomy, and confidentiality in different fields specifically focusing on medicine, law, finance, and national security. They created a benchmark and evaluated it on 12 language models to examine the performance of the different models. Findings from their study showed that the tested models performed well across different domains and that even perturbation, such as by changing the role of the model did not seem to significantly impact the results.

Critical strengths

Strengths:

  1. The paper is well written and easy to follow
  2. The topic seems well motivated
  3. The research instrument is realistic and draw from real professional documents used by the different domains being investigated
  4. Some of the findings are interesting.
Critical weaknesses

Weaknesses:

  1. The size of the scenarios (107) is too small to form a meaningful generalization
  2. I do not see much reason to include national security in this context
  3. Lack of expert-authored scenarios leads to concerns about how these findings will generalize to real-world scenarios.
Questions for authors

Can the authors get even if it is 15 to 20 expert-authored scenarios in one or two of the domains so we could have a real-world validation of the findings?

3 comments
AuthorsOpenReview ↗

Scenario coverage and inclusion of national security domain

Thank you for the constructive review and for highlighting questions about scenario coverage, the inclusion of National Security, and external professional validation. We have revised the paper and performed additional robustness analyses accordingly.

Note on corrected results: During rebuttal we identified output truncation in affected model runs, regenerated those responses with corrected output limits, and re-judged them. This changes the submitted Main-dataset mean from 95.6% to 95.9%; the pooled headline estimates change only slightly (by at most 0.3 percentage points). All results below use the corrected outputs.

1. Scenario coverage

We agree that 107 scenarios should not be interpreted as exhaustively representing the full practice space of medicine, law, finance, and national security. The Main dataset contains 10 higher-level templates and 39 distinct sub-templates: 80 Base scenarios, 24 Exception scenarios, and 3 Edge scenarios. The complete coverage is:

DomainHonestyAutonomyConfidentialityTotal
Medicine8 (5 B, 3 E)11 (5 B, 3 E, 3 Edge)8 (5 B, 3 E)27
Law5 B12 (9 B, 3 E)8 (5 B, 3 E)25
Finance5 B12 (9 B, 3 E)8 (5 B, 3 E)25
National Security9 B12 (9 B, 3 E)9 B30
Total274733107

This table has been added to the manuscript. We have also correspondingly narrowed the claims in the paper: ContextAdapt is intended as a controlled, source-grounded testbed of the normative structures represented in the selected professional sources, not a comprehensive catalogue of professional situations. Moreover, given the time constraints of the rebuttal period, significantly expanding the dataset with professionally validated scenarios was unfeasible. We therefore chose to make the current scope explicit rather than imply comprehensiveness from a modest increase in scenario count.

We also tested whether particular templates or scenario families disproportionately determine the conclusions. Removing each of the 10 Main templates in turn leaves Main appropriateness between 95.2% and 96.6%. Removing each of the 18 Secondary base-scenario families leaves the High–Low stakes gap between +7.1 and +12.3 points, while role and domain-label gaps remain at most 1.4 points. Correct-framework model rankings remain particularly stable (Spearman ρ ≥ 0.98). The small aggregate Exception–Base difference is less stable across templates. Therefore, we have revised our interpretation to avoid overgeneralising this result.

2. Why National Security?

We included National Security because it provides a deliberately different normative structure from the predominantly person-facing relationships in medicine, law, and finance. In particular, obligations such as confidentiality can be directed towards classified information, sources, methods, institutions, or oversight bodies rather than solely towards an individual counterpart. This allows the benchmark to test whether models track changes in the target of an obligation rather than simply applying a familiar person-facing interpretation.

We agree that National Security has thinner documentary coverage than the other domains. We therefore repeated the pooled analyses after excluding it:

AnalysisMain appropriatenessCorrect professional frameworkHigh − Low stakes gap
All four domains95.9%46.1%+10.3 pts
Excluding National Security96.4%53.7%+7.6 pts

These results show that the principal conclusions therefore do not depend on National Security: action-level appropriateness remains near ceiling, professional grounding remains substantially lower, and the stakes effect remains sizeable.

AuthorsOpenReview ↗

Expert-authored scenarios / professional validation

3. Expert-authored scenarios / professional validation

We agree that independent validation by qualified domain professionals would strengthen ContextAdapt, and this is a potential avenue for future work. For the current submission, no medical, legal, financial, or national-security professionals independently validated the complete Value × Domain matrix, reference outcomes, exception boundaries, or scenario set. It was not possible to obtain a credible expert-authored validation set within the rebuttal period, and we do not believe that a rushed set of 15–20 cases would constitute meaningful professional validation.

We have therefore made this limitation more prominent. ContextAdapt should be interpreted as a source-grounded testbed, rather than professionally certified ground truth, and “appropriateness” denotes correspondence with the operationalisations represented in the selected source corpus.

We have also clarified how that corpus was constructed. We identified 45 candidate instruments by reviewing major international and European professional and regulatory bodies, together with major bodies in the jurisdictions represented in the study, particularly the UK and US, and locating their professional ethics codes, conduct standards, and closely related normative guidance.

DomainCandidate instrumentsRetained
Medicine99
Law85
Finance147
National Security143
Total4524

Professional self-regulatory codes were prioritised. Where no suitable practitioner-facing code was available, or where an international or regulatory framework directly supplied a norm used in the matrix, we also retained relevant governmental, institutional, international, or regulatory frameworks.

Relevant provisions from the retained sources were used to derive the default operationalisation, target/beneficiary, and documented override conditions for each value-domain combination. The initial extraction was carried out by the first author and the resulting operationalisations checked by the co-author. This provides source grounding and author-level checking but does not substitute for independent domain-expert validation, which we now state prominently as a limitation.

To summarise, we agree that broader scenario coverage and independent professional review are important next steps, and the revised paper now distinguishes much more clearly between evidence from a controlled source-grounded benchmark and claims about comprehensive real-world professional practice.

I thank the authors for engaging with the feedback. The responses you have provided seem reasonable and makes sense for the current paper. I will keep my score as is. I look forward to the continued work of the authors in this area that engages with practitioners in real world scenarios.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.