- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
RealEstateBench evaluates seller agents in a simulated property market where sellers know defects that buyers cannot initially observe. It reports commission earnings alongside disclosure, misrepresentation, influence, and listing-integrity measures. A GPT-5 examiner is calibrated on synthetic examples and validated on trajectories reconstructed from three hundred court cases. Eight models across forty sessions show different earnings and conduct profiles, including an observed commission–compliance frontier.
The market deliberately separates information available to sellers and buyers, while the examiner has access to the relevant ground-truth records. The conduct components capture different behaviors: inadequate disclosure, contradictory factual claims, illegitimate influence and public listing integrity. The experiment fixes the seller scaffold and compares model identities, but transactions change the sequence of subsequent properties and buyer encounters. The result is therefore a comparison of complete policies in this simulated market, not a controlled estimate of the isolated economic effect of a particular conduct violation.
Critical strengths
-
The continuing market and private property facts give the paper a useful setting for studying delegated agents. Recording unsuccessful negotiations as well as sales is important, since an outcome-only evaluation would miss much of the conduct being measured. Grounding the conduct categories in professional standards gives the benchmark a more specific purpose than generic negotiation success.
The benchmark is also stronger for using enumerated private facts rather than relying only on subjective impressions of negotiation style. The difference between what an agent knows and what it communicates provides a concrete basis for evaluating disclosure and contradiction. The continuing-market design exposes behavior across multiple interactions and repeated assignments, which is useful for a professional-delegation setting. I encourage the authors to make these specific contributions central to the paper’s framing.
-
The examiner validation is more substantial than simply giving an LLM a rubric and accepting its scores. The separate case-derived corpus, component-level error rates, weighting checks and recall correction are meaningful strengths. The paper also acknowledges that influence violations can be missed and that its model comparisons are descriptive.
The component-level validation results are especially useful because they reveal uneven examiner performance. A single overall agreement statistic would obscure the poorer recall for some influence categories. The paper instead reports those differences and explores a recall correction. Its component-exclusion checks also address whether the observed frontier is entirely dependent on one part of the compliance score. These analyses are substantive work and should remain part of a fair assessment even though they do not settle all of the measurement questions.
Critical weaknesses
-
My strongest concern is the gap between the conclusions and the evidence supporting them. The paper does report session-bootstrap intervals and corrected tests. However, there are only five independent sessions per model, and the authors explicitly state that no two-sided pairwise comparison can survive their correction. The many transactions and negotiations do not supply that many independent model replications. The frontier should remain a descriptive feature of these observed means, and “avoidable noncompliance” should not imply that correcting a particular agent’s conduct would preserve its earnings. ABC (Zhu et al., 2025) also emphasizes validating an LLM judge against human judgments and using non-AI or trivial baselines to clarify what benchmark scores mean. The reconstructed court-case validation is useful, but it does not establish blinded human agreement on the actual market traces or provide a human negotiation performance anchor. A controlled honest seller and a passive seller would also help show whether favorable conduct scores require competent delegation rather than limited assessable behavior. These checks are not reported.
The session should remain the relevant independent experimental unit when presenting uncertainty. Repeated negotiations within a session share sellers, memory, property profiles and an evolving market, so treating the individual records as independent replications would not solve the five-session limitation. The current session-level approach is appropriate; my concern is how strongly its output is interpreted. Bootstrap frontier-membership frequencies conditional on five observed seeds should not be read as probabilities that the same models belong to a population frontier.
The case-derived validation and human performance questions should also remain distinct. The former tests whether an examiner reproduces labeled conduct distinctions on reconstructed examples; the latter asks what a capable human negotiator achieves in the market. Both are useful, but one cannot stand in for the other. For the immediate examiner concern, a blinded, stratified sample of actual traces is a proportionate request. For the performance claim, a human anchor or a well-specified non-AI policy would clarify whether the economic and compliance coordinates correspond to competent delegated service. The paper can acknowledge the limit if such evidence is unavailable.
-
Evaluating negotiation agents through both economic outcomes and behavioral profiles is already established. For example, PieArena: Ranking and Profiling Language Agents in Realistic Negotiation Scenarios combines explicit negotiation utilities with human comparisons and behavioral diagnostics. RealEstateBench’s distinct contribution is the professional-conduct instrument and the continuing delegated-sales environment. I would like the paper to make clearer what new capability or finding those additions establish. This connects to ABC’s recommendation to explain how measured outcomes represent the intended capability: commission measures the seller agent’s compensation, whereas “fiduciary performance” concerns the homeowner’s price and timing goals. More commission need not establish better client outcomes. The paper does not yet validate that economic proxy strongly enough to support its broader interpretation.
The continuing market and case-grounded conduct instrument could justify the paper independently of novelty in the generic joint-outcome framing. I would therefore ask the authors to demonstrate what these additions enable: a new distinction between professional behaviors, a measurement that changes conclusions relative to a simpler rubric, or an evaluation problem existing negotiation tasks cannot capture. This is a narrower and more productive claim than treating the coexistence of economic outcomes and behavioral diagnostics as new in itself.
The economic proxy is particularly important to that demonstration. The prompts give homeowners minimum-price and timing goals, but commission can also reward throughput and the agent’s own compensation. The existing market records should help distinguish these objects. The conduct aggregate also combines several dimensions and uses different exposure-sensitive aggregation rules: omission and misrepresentation take a maximum across negotiations, whereas influence uses a mean. Appendix B discusses the exposure issue and reports small simple correlations, which is useful evidence. A matched-exposure or stratified sensitivity check would nevertheless make it easier to tell whether a model’s profile reflects conduct differences or opportunities to exhibit them.
Overall Assessment
Presentation: The market, metrics, and descriptive results are clearly presented.
The separation of conduct components is helpful. I would keep the direction and aggregation of each component visible when interpreting the frontier, since a higher aggregate score need not mean better behavior on every dimension.
Informativeness: The joint economic and conduct evaluation is useful for studying delegated agents.
Soundness: The descriptive comparisons, human calibration and client-outcome interpretation leave fixable concerns.
I lean toward acceptance for the continuing-market setup and the case-grounded professional-conduct instrument. My support is for that narrower contribution. The broader framing is less novel than the paper suggests, the model frontier remains descriptive, and neither client welfare nor human-level delegated performance is established. Clearer reporting, direct validation on actual negotiations, and a convincing account of what the conduct instrument adds would strengthen my assessment.
I encourage the authors to sharpen the contribution around professional conduct in a continuing delegated market. That could be valuable even without broad superiority claims or a new theory of negotiation. The case-derived corpus and grounded simulator are a useful starting point. My remaining concerns ask whether the resulting scores support the paper’s interpretation, and whether the observed patterns add knowledge beyond a single configuration’s descriptive model comparison. Addressing those points would make the acceptance case clearer.
Questions for authors
-
Which conclusions about model differences and the frontier are supported beyond the observed means from five sessions per model? How should the economic and conduct scores be interpreted relative to human performance or controlled seller policies? On a blinded sample of actual negotiation traces, what is examiner agreement with human judgments, including variation across seller models?
-
What does the professional-conduct formulation establish beyond existing economic-outcome and behavioral-profile negotiation benchmarks such as PieArena? Can the existing market records show how commission relates to homeowner price/timing goals or client welfare, and clarify which findings depend on the new conduct instrument?
-
How do frontier membership and model profiles change under alternative reasonable compliance weights and matched interaction exposure? Which conclusions remain descriptive after accounting for the five-session sample, and which have independent support?
-
Can the existing records report homeowner goal fulfillment, such as acceptable price and timing, alongside commission? What result specifically demonstrates the additional value of the case-grounded instrument compared with a simpler factuality/disclosure rubric?
Thank you very much for taking the time to review our work and for your thoughtful feedback! As the discussion period is coming to an end, we just wanted to follow up and see if you have any further questions or concerns. We'd be happy to clarify anything.