acceptodds
Under review as a conference paper at ICLR 2027

RealEstateBench: Measuring Fiduciary Performance of AI Agents

Abstract

Task-success benchmarks ask whether an agent can complete a task; safety benchmarks ask whether it will refuse harmful requests. Neither alone establishes whether an agent honors professional obligations while pursuing its objective. To address this gap, we define fiduciary performance: how well an agent achieves the outcome it was retained for, subject to the duties of the role. We operationalize this in RealEstateBench, a long-running simulated real estate market where seller agents represent homeowner clients, market properties, and negotiate sales through structured actions and open-ended dialogue. To make conduct measurable, we derive rubrics from statute and case law, assess behavior against simulator ground truth, and validate the examiner on 300 seller–buyer trajectories restructured from court cases and verified against source facts and holdings. Across eight models and 40 sessions, compliance and commission are nearly uncorrelated (), yet five model means are dominated, showing that much of the observed noncompliance yields no economic benefit. The three nondominated models span the observed frontier from higher compliance to higher commission, revealing a remaining tradeoff between economic performance and professional conduct; the highest earner makes 55% more commission while scoring 17 percentage points lower on compliance. Models with similar earnings also differ in aggregate compliance and conduct profiles, showing that similar economic outcomes can arise from qualitatively different behavior. Together, these results show that fiduciary performance separates avoidable noncompliance from the tradeoffs that remain on the observed frontier, providing a concrete target for improving both economic performance and professional conduct.

Primary area:
datasets and benchmarks
Keywords:
Agent Evaluation, Professional Conduct, Multi-Agent Market

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XxK9OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

RealEstateBench evaluates seller agents in a simulated property market where sellers know defects that buyers cannot initially observe. It reports commission earnings alongside disclosure, misrepresentation, influence, and listing-integrity measures. A GPT-5 examiner is calibrated on synthetic examples and validated on trajectories reconstructed from three hundred court cases. Eight models across forty sessions show different earnings and conduct profiles, including an observed commission–compliance frontier.

The market deliberately separates information available to sellers and buyers, while the examiner has access to the relevant ground-truth records. The conduct components capture different behaviors: inadequate disclosure, contradictory factual claims, illegitimate influence and public listing integrity. The experiment fixes the seller scaffold and compares model identities, but transactions change the sequence of subsequent properties and buyer encounters. The result is therefore a comparison of complete policies in this simulated market, not a controlled estimate of the isolated economic effect of a particular conduct violation.

Critical strengths
  1. The continuing market and private property facts give the paper a useful setting for studying delegated agents. Recording unsuccessful negotiations as well as sales is important, since an outcome-only evaluation would miss much of the conduct being measured. Grounding the conduct categories in professional standards gives the benchmark a more specific purpose than generic negotiation success.

    The benchmark is also stronger for using enumerated private facts rather than relying only on subjective impressions of negotiation style. The difference between what an agent knows and what it communicates provides a concrete basis for evaluating disclosure and contradiction. The continuing-market design exposes behavior across multiple interactions and repeated assignments, which is useful for a professional-delegation setting. I encourage the authors to make these specific contributions central to the paper’s framing.

  2. The examiner validation is more substantial than simply giving an LLM a rubric and accepting its scores. The separate case-derived corpus, component-level error rates, weighting checks and recall correction are meaningful strengths. The paper also acknowledges that influence violations can be missed and that its model comparisons are descriptive.

    The component-level validation results are especially useful because they reveal uneven examiner performance. A single overall agreement statistic would obscure the poorer recall for some influence categories. The paper instead reports those differences and explores a recall correction. Its component-exclusion checks also address whether the observed frontier is entirely dependent on one part of the compliance score. These analyses are substantive work and should remain part of a fair assessment even though they do not settle all of the measurement questions.

Critical weaknesses
  1. My strongest concern is the gap between the conclusions and the evidence supporting them. The paper does report session-bootstrap intervals and corrected tests. However, there are only five independent sessions per model, and the authors explicitly state that no two-sided pairwise comparison can survive their correction. The many transactions and negotiations do not supply that many independent model replications. The frontier should remain a descriptive feature of these observed means, and “avoidable noncompliance” should not imply that correcting a particular agent’s conduct would preserve its earnings. ABC (Zhu et al., 2025) also emphasizes validating an LLM judge against human judgments and using non-AI or trivial baselines to clarify what benchmark scores mean. The reconstructed court-case validation is useful, but it does not establish blinded human agreement on the actual market traces or provide a human negotiation performance anchor. A controlled honest seller and a passive seller would also help show whether favorable conduct scores require competent delegation rather than limited assessable behavior. These checks are not reported.

    The session should remain the relevant independent experimental unit when presenting uncertainty. Repeated negotiations within a session share sellers, memory, property profiles and an evolving market, so treating the individual records as independent replications would not solve the five-session limitation. The current session-level approach is appropriate; my concern is how strongly its output is interpreted. Bootstrap frontier-membership frequencies conditional on five observed seeds should not be read as probabilities that the same models belong to a population frontier.

    The case-derived validation and human performance questions should also remain distinct. The former tests whether an examiner reproduces labeled conduct distinctions on reconstructed examples; the latter asks what a capable human negotiator achieves in the market. Both are useful, but one cannot stand in for the other. For the immediate examiner concern, a blinded, stratified sample of actual traces is a proportionate request. For the performance claim, a human anchor or a well-specified non-AI policy would clarify whether the economic and compliance coordinates correspond to competent delegated service. The paper can acknowledge the limit if such evidence is unavailable.

  2. Evaluating negotiation agents through both economic outcomes and behavioral profiles is already established. For example, PieArena: Ranking and Profiling Language Agents in Realistic Negotiation Scenarios combines explicit negotiation utilities with human comparisons and behavioral diagnostics. RealEstateBench’s distinct contribution is the professional-conduct instrument and the continuing delegated-sales environment. I would like the paper to make clearer what new capability or finding those additions establish. This connects to ABC’s recommendation to explain how measured outcomes represent the intended capability: commission measures the seller agent’s compensation, whereas “fiduciary performance” concerns the homeowner’s price and timing goals. More commission need not establish better client outcomes. The paper does not yet validate that economic proxy strongly enough to support its broader interpretation.

    The continuing market and case-grounded conduct instrument could justify the paper independently of novelty in the generic joint-outcome framing. I would therefore ask the authors to demonstrate what these additions enable: a new distinction between professional behaviors, a measurement that changes conclusions relative to a simpler rubric, or an evaluation problem existing negotiation tasks cannot capture. This is a narrower and more productive claim than treating the coexistence of economic outcomes and behavioral diagnostics as new in itself.

    The economic proxy is particularly important to that demonstration. The prompts give homeowners minimum-price and timing goals, but commission can also reward throughput and the agent’s own compensation. The existing market records should help distinguish these objects. The conduct aggregate also combines several dimensions and uses different exposure-sensitive aggregation rules: omission and misrepresentation take a maximum across negotiations, whereas influence uses a mean. Appendix B discusses the exposure issue and reports small simple correlations, which is useful evidence. A matched-exposure or stratified sensitivity check would nevertheless make it easier to tell whether a model’s profile reflects conduct differences or opportunities to exhibit them.

Overall Assessment

Presentation: The market, metrics, and descriptive results are clearly presented.

The separation of conduct components is helpful. I would keep the direction and aggregation of each component visible when interpreting the frontier, since a higher aggregate score need not mean better behavior on every dimension.

Informativeness: The joint economic and conduct evaluation is useful for studying delegated agents.

Soundness: The descriptive comparisons, human calibration and client-outcome interpretation leave fixable concerns.

I lean toward acceptance for the continuing-market setup and the case-grounded professional-conduct instrument. My support is for that narrower contribution. The broader framing is less novel than the paper suggests, the model frontier remains descriptive, and neither client welfare nor human-level delegated performance is established. Clearer reporting, direct validation on actual negotiations, and a convincing account of what the conduct instrument adds would strengthen my assessment.

I encourage the authors to sharpen the contribution around professional conduct in a continuing delegated market. That could be valuable even without broad superiority claims or a new theory of negotiation. The case-derived corpus and grounded simulator are a useful starting point. My remaining concerns ask whether the resulting scores support the paper’s interpretation, and whether the observed patterns add knowledge beyond a single configuration’s descriptive model comparison. Addressing those points would make the acceptance case clearer.

Questions for authors
  1. Which conclusions about model differences and the frontier are supported beyond the observed means from five sessions per model? How should the economic and conduct scores be interpreted relative to human performance or controlled seller policies? On a blinded sample of actual negotiation traces, what is examiner agreement with human judgments, including variation across seller models?

  2. What does the professional-conduct formulation establish beyond existing economic-outcome and behavioral-profile negotiation benchmarks such as PieArena? Can the existing market records show how commission relates to homeowner price/timing goals or client welfare, and clarify which findings depend on the new conduct instrument?

  3. How do frontier membership and model profiles change under alternative reasonable compliance weights and matched interaction exposure? Which conclusions remain descriptive after accounting for the five-session sample, and which have independent support?

  4. Can the existing records report homeowner goal fulfillment, such as acceptable price and timing, alongside commission? What result specifically demonstrates the additional value of the case-grounded instrument compared with a simpler factuality/disclosure rubric?

1 comment

Thank you very much for taking the time to review our work and for your thoughtful feedback! As the discussion period is coming to an end, we just wanted to follow up and see if you have any further questions or concerns. We'd be happy to clarify anything.

Reviewer_JTRaOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes fiduciary performance to evaluate whether AI agents achieve retained outcomes while meeting role obligations. It introduces RealEstateBench, a simulated real estate market where seller agents market and negotiate under information asymmetry and conduct requirements. Agents are evaluated on economic performance (commissions) and a legally grounded Compliance Score covering disclosure, misrepresentation, influence tactics, and listing integrity. Across eight models and repeated sessions, commission and compliance are nearly uncorrelated; several models are dominated, while nondominated models still show an economic-conduct tradeoff. Similar economic outcomes can mask different conduct profiles, motivating component-level evaluation.

Critical strengths
  1. The paper identifies a meaningful gap in existing agent evaluation: task-success benchmarks primarily measure whether an agent achieves its objective, while safety benchmarks focus on whether an agent avoids harmful behavior. Neither directly evaluates whether an agent fulfills the professional obligations associated with a delegated role. The proposed concept of fiduciary performance provides a principled framework for jointly considering task-related economic outcomes and role-specific professional conduct.

  2. The benchmark goes beyond ad-hoc behavioral rubrics by grounding its four conduct dimensions in California statutes and case law. More importantly, the LLM-based examiner is explicitly validated rather than treated as a black-box evaluator: the authors construct a 300-trajectory validation corpus from real court cases and compare examiner judgments against reference labels derived from the underlying facts and holdings. The resulting agreement provides meaningful evidence that the proposed conduct metrics can be evaluated with reasonable consistency.

Critical weaknesses
  1. Although the paper observes clear differences in mean commission and compliance, only five sessions per model are evaluated, and no pairwise comparison survives multiple-comparison correction. This makes the reported dominance relationships and frontier membership difficult to distinguish from sampling variability.

  2. The GPT-5 examiner is validated against court-case-derived reference labels, but it shares a model family with GPT-5.5, one of the evaluated models. While this does not establish systematic bias, an examiner from a disjoint model family would provide stronger evidence that the reported conduct differences are not examiner-dependent.

Questions for authors
  1. How robust are the reported frontier and dominance relationships to sampling variability? Given the limited number of sessions and the lack of significant pairwise comparisons after correction, can the authors provide additional evidence that these relationships are not driven by the particular sampled trajectories?

  2. Does examiner choice affect the reported compliance results? In particular, would the main conclusions remain unchanged when using an examiner from a disjoint model family?

Reviewer_jENqOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a simulated real-estate sales benchmark that evaluates agents jointly on economic outcomes and professional conduct. It derives conduct rubrics from legal sources, evaluates eight models across 40 sessions, and validates an LLM examiner against 300 court-case-derived trajectories. The main insight is that task outcome alone may conceal differences in how an agent achieves that outcome.

Critical strengths
  1. Somewhat meaningful evaluation perspective. Evaluating economic outcomes alongside professional conduct offers some value for delegated agents beyond task success alone.

  2. Legally grounded evaluation. The use of legal rubrics and case-derived reference labels provides some credibility to the evaluation framework.

Critical weaknesses
  1. Limited novelty and technical contribution. The benchmark is small-scale and focuses on a single real estate task with little diversity. The technical implementation appears relatively straightforward, mainly combining simulation, legal rubrics, and LLM-based judging. I do not see substantial technical novelty or a particularly challenging benchmark design.

  2. Lack of realism and broader significance. Despite focusing on real estate transactions, the benchmark relies on simulated data and synthetic agent interactions rather than real-world transactions. I find the setting overly simplified and the conclusions too specific to this narrow task. It is unclear what generalizable insights this work offers to the broader agent community. Moreover, commission alone does not adequately represent the client's actual objectives.

  3. Insufficient evaluation reliability and robustness. The LLM examiner exhibits uneven reliability across conduct categories (e.g., κ = 0.64 for emotional exploitation), potentially introducing systematic biases into compliance measurements. Although the paper includes limited sensitivity analyses, it remains unclear whether the model rankings and Pareto frontier are robust to alternative component weights, examiner errors, or independent human judgments on unseen interactions.

Questions for authors

see weakness.

1 comment

Thank you very much for taking the time to review our work and for your thoughtful feedback! As the discussion period is coming to an end, we just wanted to follow up and see if you have any further questions or concerns. We'd be happy to clarify anything.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.