- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper introduces ENURI, a benchmark for evaluating how multimodal agents recognize and use visual evidence in bilateral price negotiations. Building on TERMS-Bench, the authors introduce synthetic product defects and organize them into four conditions based on Visibility (Obvious/Subtle) and Alignment (Aligned/Misaligned). The benchmark evaluates negotiation outcomes, defect citations, and price adjustments against predefined defect impacts. The authors evaluate 13 LLM/VLM agents and three rule-based baselines across four price-impact mappings, totaling 6,400 negotiation episodes. The results suggest that Detection and Valuation exhibit different failure patterns, while terminal negotiation outcomes do not fully reflect how agents use visual evidence.
Critical strengths
1.The paper introduces an interesting evaluation setting by combining controlled visual defects with interactive price negotiation. The Visibility × Alignment design aims to examine visual evidence along two complementary axes, while the structured defect citation interface enables direct recording and analysis of how agents use evidence during negotiation. 2.The evaluation includes 13 LLM/VLM agents from different model families and capability levels, together with three rule-based baselines. The authors conduct 6,400 negotiation episodes under four price-impact mappings. The differences between terminal outcomes and citation behavior, together with the observed tendency toward insufficient price adjustments, provide useful insights. Detailed experimental configurations and negotiation traces also improve transparency.
Critical weaknesses
1.The proposed metrics do not sufficiently establish the claimed separation between Detection and Valuation failures. Defect citations reflect not only visual perception but also strategic disclosure, while concessions made during negotiation do not necessarily represent an agent's valuation of a defect. Although the authors acknowledge the former limitation and provide separate analyses for buyers and sellers, the lack of independent perception and valuation assessments leaves the central claim insufficiently validated. 2.The key labels of the synthetic dataset lack sufficient independent validation. Visibility and Alignment are assigned through generation instructions, while the QA procedure only requires identifying some visible damage without verifying the intended defect or whether it satisfies the assigned conditions. Furthermore, using the same model for image generation and QA may introduce shared biases. These issues raise concerns about the reliability of the ground truth used to support the main findings.
Questions for authors
- The authors acknowledge that citation recall may reflect strategic disclosure in addition to Detection, and provide separate analyses for buyers and sellers. Could the authors provide an independent perception assessment to further validate citation recall as a measure of Detection?
- Concessions made when citing a defect may depend on previous offers and negotiation strategies rather than solely on the agent's Valuation. Could the authors provide an independent valuation assessment or an oracle experiment to validate whether the proposed metric reliably captures Valuation failures?
- Visibility and Alignment labels are assigned during image generation, while the current QA procedure does not explicitly verify whether the generated defects satisfy these conditions. Could the authors provide independent validation, such as human annotation or evaluation using other model families, to assess the reliability of these labels?
5 comments
Rebuttal by Authors
W1 / Q2
Could the authors provide an independent valuation assessment or an oracle experiment to validate whether the proposed metric reliably captures Valuation failures?
Thank you for suggesting these two experiments, and we ran both. The submitted metric used the concession at a citing turn, but under mapping B, in 338 of the 347 turns in which a buyer cited a defect, the buyer raised its own offer as in any other turn, so the metric could not separate the defect from the flow of the negotiation. We therefore compared prices before the negotiation can intervene.
For an independent Valuation measure, we reran each episode with the original defect-free photo and compared the agent's first offer, made before the counterpart can react to a citation. Item, role, reservation price, counterpart, and seed are the same, so earlier offers or the course of the negotiation cannot produce the difference. For the oracle experiment, we added one sentence naming the defect type to the same defect photo. Since agents do not know the price mapping, we score by tier order rather than by dollar amount.
| First-offer drop (% of listing price) | Cosmetic | Structural | Structural minus cosmetic (points) | Agents with a significant order |
|---|---|---|---|---|
| Defect photo only | 1.8 | 1.9 | 0.1 [-0.6, 0.7] | 0/11 |
| Defect type also named (oracle) | 1.3 | 2.3 | 1.0 [0.4, 1.6] | 6/11 |
From the photo alone, agents lower their first offer by 1.8% on the defect photo (all 11 agents), but they discount cosmetic and structural defects almost alike and follow visibility instead (2.4 to 2.9% in the Obvious cells, 0.6 to 0.7% in the Subtle cells). Human discounts behave the same way. Naming the type produces a tier order with larger discounts for structural defects, significant for 6 of 11 agents. Agents did use the information. Citation of the true defect type rose from 62.5% to 99.7% for buyers and from 38.8% to 76.0% for sellers.
The ordering failure therefore arises mainly when reading the defect type from the photo, and it was the oracle experiment you suggested that located this stage. The total discount stays almost the same (1.8% and 1.9%) and is only redistributed by severity, so there is no basis for attributing the under-adjustment in Table 14 of the submission (Table 13 in the revision) to a perception failure. Final deal prices are also 0.7% lower with the defect photo, so the effect persists to the end of the negotiation.
We revised §3.5, the Valuation paragraph of §7 with Figures 6B and 6C, and Appendices I.3 and I.5.
Rebuttal by Authors
W2 / Q3
Visibility and Alignment labels are assigned during image generation, while the current QA procedure does not explicitly verify whether the generated defects satisfy these conditions. Could the authors provide independent validation, such as human annotation or evaluation using other model families, to assess the reliability of these labels?
As you suggested, we added both human annotation and judgments by models of other families, and the Visibility label held, while raters named the defect type correctly for about two thirds of the defects they found. The Alignment label was not rated directly, and it disagrees with the price tier for 58 of the 270 defect items, so we state the Valuation conclusions by price tier instead of by this label. The original QA showed only the photo and asked the checker to find the damage again, so that it would not lean toward confirming the label (Appendix B.4), but this could not verify the labels. Ten graduate students who were not told that the study evaluates AI agents rated the 100 photos of the main runs with three raters per photo, and Claude Fable 5.2 and GPT-6.1 Sol, from families other than the Gemini generator, judged all 324 photos.
The Visibility label holds. Humans found Subtle defects 29.8 points [17.6, 42.4] less often than Obvious ones (98.6% against 68.9%), and Fable shows a similar gap. Agreement between the rated noticeability and the label is lower, at 79.7% (κ = 0.44), because raters who did find a Subtle defect often called it easy to see. A rater who has found a defect already knows where it is, so we measure visibility by whether a defect is found.
The defect type can be read from the photo for about two thirds of the found defects. Humans named the type correctly for 65.7% of the defects they found (κ = 0.73 among raters), and the type judgments of Fable agree with the human majority at κ = 0.72. Humans named scratches correctly in only 27% of cases and Fable in 48%, often reading them as a crack or a tear, which lie in a different price tier, and we report this in the paper. Sol reports a flaw on 75% of defect-free photos, a threshold too loose to use, so we rely on Fable.
In the table, OA, OM, SA, and SM denote Obvious-Aligned, Obvious-Misaligned, Subtle-Aligned, and Subtle-Misaligned.
| Found, OA | Found, OM | Found, SA | Found, SM | Flaw reported on defect-free photos | Type correct among found | |
|---|---|---|---|---|---|---|
| Humans (100 photos) | 100.0% | 97.7% | 66.7% | 70.6% | 7.9% | 65.7% |
| Claude Fable 5.2 (324 photos) | 98.3% | 98.8% | 71.2% | 76.8% | 17.9% | 65.6% |
| GPT-6.1 Sol (324 photos) | 99.6% | 98.3% | 93.6% | 97.2% | 75.3% | 65.8% |
The Alignment label follows the generation instruction, and it agrees with the price tier of the defect type for 212 of the 270 defect items (59 of the 79 in the evaluation sample). We now state this in the paper. The Valuation conclusions rest on analyses by price tier (photo swap, oracle, comparison within structural defects), which do not use the Alignment label.
The Gemini agent, which shares a family with the generator, does not see Subtle defects better (private-field drop of 19.6 points as buyer and 16.9 as seller, against 11-agent averages of 16.9 and 18.7, and a buyer citation drop of 17.0 points in the runs without the field, against an average of 11.9). Removing four photos with possible rendering problems leaves the pooled results nearly unchanged (see our response to Reviewer 7YLZ, Q3, which also gives the one per-agent count that changes).
We revised §3.2, §3.6, §8, and Appendix I.4. The human rating data will be released with the dataset.
Your specific requests made the evidence for our central claims much firmer. Thank you.
Rebuttal by Authors
Thank you for your review. We added all the independent assessments you requested. Several results overlap with the questions of Reviewer 7YLZ, and in places we point to that response for details.
W1 / Q1
We added the independent perception assessment you asked for. It shows that buyer citation recall is close to perception on average, but seller citation recall is not. To measure perception apart from citation, we now add a private field that never reaches the counterpart, in which the agent records the defects it sees regardless of any incentive to cite them. Nine of the 11 agents wrote this field before the public message in at least 89% of turns. Gemini 3.1 Pro did so in 49% of turns, and for ThinkingMachines-Inkling the order is not observable (Appendix I.2).
In the runs with the field, buyers cite 99% of the defects they record there. Perception, however, differs between buyer and seller by only 3.5 points, while citation differs by 23.6 points, and this citation gap ranges from 3 to 56 points across agents. Seller citations thus include a disclosure decision.
The Subtle drop remains in the private field (16.9 points for buyers, 18.7 for sellers), significant for 7 of 11 agents as buyer and 9 of 11 as seller. Adding the field lowered seller citation by 12.2 points, so we measure perception in the runs with the field and compare citation by role in the runs without it. The drop also remains after subtracting how often each agent lists the same type on defect-free photos, and when only the first turn is used (see our response to Reviewer 7YLZ, W1 / Q1).
Detection is now defined by the private field, and citation recall is reported separately as what the agent chooses to say.
We revised §3.5, the Detection paragraph of §7, and Appendix I.2.