acceptodds
Under review as a conference paper at ICLR 2027

ENURI: Evidence Negotiation Using Reasoning over Images

Abstract

A language-model agent acting on visual evidence must notice the evidence that matters (Detection) and weigh it correctly (Valuation), but outcome-level evaluations score only the final result and cannot tell which of the two failed. We introduce ENURI, a bilateral price-negotiation benchmark that extends the diagnostic TERMS-Bench environment with an image channel in which an item's defects appear only in its photo and carry a known ground-truth price impact. Placing defects on a visibility × alignment grid (Obvious/Subtle × Aligned/Misaligned) makes the correct response to evidence verifiable, so every failure can be attributed to one stage or the other. We evaluate 13 frontier, open-weight, and sub-frontier LLM/VLM agents and three rule-based baselines over 6,400 episodes, with rankings stable across four price-impact mappings. The two stages fail along different axes, and neither failure is visible in the outcome score. Detection recall falls from 0.551 to 0.278 on Subtle defects for all 13 agents, whereas Valuation accuracy holds between 0.92 and 0.99 in three cells of the grid and falls to 0.73 in Subtle-Misaligned alone. Claude Opus 4.6 captures the most surplus of any agent (normalized surplus 0.593) yet shows the largest Detection drop, from 0.612 recall on Obvious defects to 0.217 on Subtle ones, while GPT-4o-mini cites most precisely yet scores the weakest overall. ENURI thus shows not only which agents fail but which stage fails. We release the benchmark, dataset, and per-episode traces.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, benchmark, evaluation, vision-language models, negotiation, multimodal negotiation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_w7doOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper introduces ENURI, a benchmark for evaluating how multimodal agents recognize and use visual evidence in bilateral price negotiations. Building on TERMS-Bench, the authors introduce synthetic product defects and organize them into four conditions based on Visibility (Obvious/Subtle) and Alignment (Aligned/Misaligned). The benchmark evaluates negotiation outcomes, defect citations, and price adjustments against predefined defect impacts. The authors evaluate 13 LLM/VLM agents and three rule-based baselines across four price-impact mappings, totaling 6,400 negotiation episodes. The results suggest that Detection and Valuation exhibit different failure patterns, while terminal negotiation outcomes do not fully reflect how agents use visual evidence.

Critical strengths

1.The paper introduces an interesting evaluation setting by combining controlled visual defects with interactive price negotiation. The Visibility × Alignment design aims to examine visual evidence along two complementary axes, while the structured defect citation interface enables direct recording and analysis of how agents use evidence during negotiation. 2.The evaluation includes 13 LLM/VLM agents from different model families and capability levels, together with three rule-based baselines. The authors conduct 6,400 negotiation episodes under four price-impact mappings. The differences between terminal outcomes and citation behavior, together with the observed tendency toward insufficient price adjustments, provide useful insights. Detailed experimental configurations and negotiation traces also improve transparency.

Critical weaknesses

1.The proposed metrics do not sufficiently establish the claimed separation between Detection and Valuation failures. Defect citations reflect not only visual perception but also strategic disclosure, while concessions made during negotiation do not necessarily represent an agent's valuation of a defect. Although the authors acknowledge the former limitation and provide separate analyses for buyers and sellers, the lack of independent perception and valuation assessments leaves the central claim insufficiently validated. 2.The key labels of the synthetic dataset lack sufficient independent validation. Visibility and Alignment are assigned through generation instructions, while the QA procedure only requires identifying some visible damage without verifying the intended defect or whether it satisfies the assigned conditions. Furthermore, using the same model for image generation and QA may introduce shared biases. These issues raise concerns about the reliability of the ground truth used to support the main findings.

Questions for authors
  1. The authors acknowledge that citation recall may reflect strategic disclosure in addition to Detection, and provide separate analyses for buyers and sellers. Could the authors provide an independent perception assessment to further validate citation recall as a measure of Detection?
  2. Concessions made when citing a defect may depend on previous offers and negotiation strategies rather than solely on the agent's Valuation. Could the authors provide an independent valuation assessment or an oracle experiment to validate whether the proposed metric reliably captures Valuation failures?
  3. Visibility and Alignment labels are assigned during image generation, while the current QA procedure does not explicitly verify whether the generated defects satisfy these conditions. Could the authors provide independent validation, such as human annotation or evaluation using other model families, to assess the reliability of these labels?
5 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for your review. We added all the independent assessments you requested. Several results overlap with the questions of Reviewer 7YLZ, and in places we point to that response for details.

W1 / Q1

Could the authors provide an independent perception assessment to further validate citation recall as a measure of Detection?

We added the independent perception assessment you asked for. It shows that buyer citation recall is close to perception on average, but seller citation recall is not. To measure perception apart from citation, we now add a private field that never reaches the counterpart, in which the agent records the defects it sees regardless of any incentive to cite them. Nine of the 11 agents wrote this field before the public message in at least 89% of turns. Gemini 3.1 Pro did so in 49% of turns, and for ThinkingMachines-Inkling the order is not observable (Appendix I.2).

ObviousSubtleDrop (points)
Perception (private field), buyer71.6%54.7%16.9
Perception (private field), seller68.9%50.2%18.7
Citation, buyer67.4%55.4%11.9
Citation, seller47.3%26.7%20.6

In the runs with the field, buyers cite 99% of the defects they record there. Perception, however, differs between buyer and seller by only 3.5 points, while citation differs by 23.6 points, and this citation gap ranges from 3 to 56 points across agents. Seller citations thus include a disclosure decision.

The Subtle drop remains in the private field (16.9 points for buyers, 18.7 for sellers), significant for 7 of 11 agents as buyer and 9 of 11 as seller. Adding the field lowered seller citation by 12.2 points, so we measure perception in the runs with the field and compare citation by role in the runs without it. The drop also remains after subtracting how often each agent lists the same type on defect-free photos, and when only the first turn is used (see our response to Reviewer 7YLZ, W1 / Q1).

Detection is now defined by the private field, and citation recall is reported separately as what the agent chooses to say.

We revised §3.5, the Detection paragraph of §7, and Appendix I.2.

AuthorsOpenReview ↗

Rebuttal by Authors

W1 / Q2

Could the authors provide an independent valuation assessment or an oracle experiment to validate whether the proposed metric reliably captures Valuation failures?

Thank you for suggesting these two experiments, and we ran both. The submitted metric used the concession at a citing turn, but under mapping B, in 338 of the 347 turns in which a buyer cited a defect, the buyer raised its own offer as in any other turn, so the metric could not separate the defect from the flow of the negotiation. We therefore compared prices before the negotiation can intervene.

For an independent Valuation measure, we reran each episode with the original defect-free photo and compared the agent's first offer, made before the counterpart can react to a citation. Item, role, reservation price, counterpart, and seed are the same, so earlier offers or the course of the negotiation cannot produce the difference. For the oracle experiment, we added one sentence naming the defect type to the same defect photo. Since agents do not know the price mapping, we score by tier order rather than by dollar amount.

First-offer drop (% of listing price)CosmeticStructuralStructural minus cosmetic (points)Agents with a significant order
Defect photo only1.81.90.1 [-0.6, 0.7]0/11
Defect type also named (oracle)1.32.31.0 [0.4, 1.6]6/11

From the photo alone, agents lower their first offer by 1.8% on the defect photo (all 11 agents), but they discount cosmetic and structural defects almost alike and follow visibility instead (2.4 to 2.9% in the Obvious cells, 0.6 to 0.7% in the Subtle cells). Human discounts behave the same way. Naming the type produces a tier order with larger discounts for structural defects, significant for 6 of 11 agents. Agents did use the information. Citation of the true defect type rose from 62.5% to 99.7% for buyers and from 38.8% to 76.0% for sellers.

The ordering failure therefore arises mainly when reading the defect type from the photo, and it was the oracle experiment you suggested that located this stage. The total discount stays almost the same (1.8% and 1.9%) and is only redistributed by severity, so there is no basis for attributing the under-adjustment in Table 14 of the submission (Table 13 in the revision) to a perception failure. Final deal prices are also 0.7% lower with the defect photo, so the effect persists to the end of the negotiation.

We revised §3.5, the Valuation paragraph of §7 with Figures 6B and 6C, and Appendices I.3 and I.5.

AuthorsOpenReview ↗

Rebuttal by Authors

W2 / Q3

Visibility and Alignment labels are assigned during image generation, while the current QA procedure does not explicitly verify whether the generated defects satisfy these conditions. Could the authors provide independent validation, such as human annotation or evaluation using other model families, to assess the reliability of these labels?

As you suggested, we added both human annotation and judgments by models of other families, and the Visibility label held, while raters named the defect type correctly for about two thirds of the defects they found. The Alignment label was not rated directly, and it disagrees with the price tier for 58 of the 270 defect items, so we state the Valuation conclusions by price tier instead of by this label. The original QA showed only the photo and asked the checker to find the damage again, so that it would not lean toward confirming the label (Appendix B.4), but this could not verify the labels. Ten graduate students who were not told that the study evaluates AI agents rated the 100 photos of the main runs with three raters per photo, and Claude Fable 5.2 and GPT-6.1 Sol, from families other than the Gemini generator, judged all 324 photos.

The Visibility label holds. Humans found Subtle defects 29.8 points [17.6, 42.4] less often than Obvious ones (98.6% against 68.9%), and Fable shows a similar gap. Agreement between the rated noticeability and the label is lower, at 79.7% (κ = 0.44), because raters who did find a Subtle defect often called it easy to see. A rater who has found a defect already knows where it is, so we measure visibility by whether a defect is found.

The defect type can be read from the photo for about two thirds of the found defects. Humans named the type correctly for 65.7% of the defects they found (κ = 0.73 among raters), and the type judgments of Fable agree with the human majority at κ = 0.72. Humans named scratches correctly in only 27% of cases and Fable in 48%, often reading them as a crack or a tear, which lie in a different price tier, and we report this in the paper. Sol reports a flaw on 75% of defect-free photos, a threshold too loose to use, so we rely on Fable.

In the table, OA, OM, SA, and SM denote Obvious-Aligned, Obvious-Misaligned, Subtle-Aligned, and Subtle-Misaligned.

Found, OAFound, OMFound, SAFound, SMFlaw reported on defect-free photosType correct among found
Humans (100 photos)100.0%97.7%66.7%70.6%7.9%65.7%
Claude Fable 5.2 (324 photos)98.3%98.8%71.2%76.8%17.9%65.6%
GPT-6.1 Sol (324 photos)99.6%98.3%93.6%97.2%75.3%65.8%

The Alignment label follows the generation instruction, and it agrees with the price tier of the defect type for 212 of the 270 defect items (59 of the 79 in the evaluation sample). We now state this in the paper. The Valuation conclusions rest on analyses by price tier (photo swap, oracle, comparison within structural defects), which do not use the Alignment label.

The Gemini agent, which shares a family with the generator, does not see Subtle defects better (private-field drop of 19.6 points as buyer and 16.9 as seller, against 11-agent averages of 16.9 and 18.7, and a buyer citation drop of 17.0 points in the runs without the field, against an average of 11.9). Removing four photos with possible rendering problems leaves the pooled results nearly unchanged (see our response to Reviewer 7YLZ, Q3, which also gives the one per-agent count that changes).

We revised §3.2, §3.6, §8, and Appendix I.4. The human rating data will be released with the dataset.

Your specific requests made the evidence for our central claims much firmer. Thank you.

Reviewer_7YLZOpenReview ↗
Rating
3(+1)
Confidence
1(−2)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ENURI, a bilateral price-negotiation benchmark extending TERMS-Bench [1] with a synthetic visual channel. Using CraigslistBargain listings [2], the authors prompt an image-editing model to inject visual defects across a 2×22 \times 2 grid of Visibility (Obvious vs. Subtle) and Alignment (Aligned vs. Misaligned), assigning hidden ground-truth price impacts under four mappings. Evaluating 13 VLM agents and 3 rule-based baselines over 6,400 episodes against a simulated counterpart, the paper aims to disentangle whether suboptimal negotiation outcomes stem from failing to notice a visual defect (Detection) or failing to adjust the price appropriately after citing it (Valuation).

Critical strengths

Attempting to separate perceptual grounding errors from downstream decision-making and valuation errors in multimodal agents is a well-motivated direction. Extending programmatic negotiation environments [1, 3] with an image modality is a reasonable setup for studying multimodal bargaining.

Critical weaknesses

The benchmark fundamentally conflates visual perception (Detection) with strategic disclosure incentives. Detection recall is measured solely by whether the agent populates the cited_defect_ids field during negotiation. However, in the counterpart kernel (Eq. 1), citing a defect carries a negative sign (sA=−1s_A = -1) when the agent plays the seller, directly penalizing the seller's offer acceptance probability and surplus. A rational seller has a strong strategic incentive to withhold defects it sees in the image. Pooling buyer and seller episodes to claim a perceptual drop—such as criticizing the top-performing agent for a large recall drop when its buyer-role recall is actually much higher than its seller-role recall in Tables 15 and 16—confounds strategic withholding with visual blindness. Indeed, when restricted to buyer-only episodes where citing is aligned with the agent's incentive, the visibility drop is statistically significant for only 5 out of 13 models (Section 7 and Figure 19).The Valuation task and the Subtle-Misaligned regime are ill-posed because the "ground-truth" price impact is unobservable from pixels.

Questions for authors
  1. Since citing a defect as a seller directly hurts the agent in Eq. 1 (sA=−1s_A = -1), how can cite-cov on seller episodes be interpreted as visual Detection rather than rational strategic withholding? Why not measure perception via a private, non-communicated state belief field before the public negotiation message is sent?
  2. In the Subtle-Misaligned condition, the image only shows a minor visual blemish while the structural severity and exact percentage mappings (A–D) are hidden from the agent. How is an agent supposed to infer the ground-truth dollar deduction from pixels alone, and how do you reconcile the claim that Valuation succeeds in Aligned quadrants (Figure 6B) with Table 14 showing pervasive under-adjustment across both Aligned and Misaligned turns?
  3. Given that the same model family generates and blind-verifies the synthetic defects without human validation (accepting any non-NONE response regardless of whether the rendered defect matches the target category or visibility), did you conduct any human evaluation to verify that Subtle-Misaligned defects are actually distinguishable and properly rendered?

[1] TERMS-bench: Diagnosing LLM Negotiation Agents Beyond Deal Rate

[2] Decoupling Strategy and Generation in Negotiation Dialogues

[3] How Well Can LLMs Negotiate? NegotiationArena Platform and Analysis

6 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for the detailed review. All three points go to the core of the measurement, and we adopted the private belief field you suggested.

W1 / Q1

... how can cite-cov on seller episodes be interpreted as visual Detection rather than rational strategic withholding? Why not measure perception via a private, non-communicated state belief field before the public negotiation message is sent?

On seller episodes, citation recall reflects both what the agent saw and what it chose to say, so it does not isolate Detection. We now define Detection by the private field you proposed and report citation recall separately. With the two separated, agents cite Subtle defects less often mostly because they see them less often, and sellers cite less than buyers because they withhold defects they have seen. The submission noted in §8 that disclosure decisions can enter seller citations and reported results by role in Tables 15 and 16 (Tables 14 and 15 in the revision) and Figure 19. The new field separates the two directly.

In every turn the agent writes the defects it sees into observed_defect_ids, which reaches neither the counterpart nor the kernel. Nine of the 11 agents wrote this field before the public message in at least 89% of turns. Gemini 3.1 Pro did so in 49% of turns, and for ThinkingMachines-Inkling the order is not observable (Appendix I.2). We played all 324 items once as buyer and once as seller, and also played the 270 defect items in both roles without the field.

ObviousSubtleDrop (points)
Perception (private field), buyer71.6%54.7%16.9
Perception (private field), seller68.9%50.2%18.7
Citation, buyer67.4%55.4%11.9
Citation, seller47.3%26.7%20.6

Even in the private field, where there is no reason to hide anything, Subtle defects are seen 17 to 19 points less often in both roles. The result holds after subtracting how often each agent lists the same type on defect-free photos, and when only the first turn is used. The citation drop on Subtle defects therefore comes mostly from seeing less. The role gap is different in kind. Perception differs between buyer and seller by only 3.5 points, while citation differs by 23.6 points. Sellers say less about what they have seen, and how much less ranges from 3 to 56 points across agents.

The buyer-only result of the submission (5/13) had wide intervals because each agent had only about 38 buyer episodes with a defect (buyer drop under mapping B 15.7 points, 95% CI [-6.2, 37.9]). The intervals were too wide to tell whether there was a drop. When all 270 items are played in both roles, buyer citation falls by 11.9 points [3.7, 20.1] on Subtle defects, significant for 7 of 11 agents, three of which sit at the threshold with an adjusted p of 0.044. Without four photos that may be poorly rendered, 4 of 11 remain significant (see Q3).

We also checked Opus 4.6 with the private field. Its perception drop (18.2 points as buyer, 16.5 as seller) is similar to that of the other agents. As a seller it cites far less than as a buyer (a gap of 34.8 points), which enlarged its pooled drop in the submission. We rewrote the Opus sentence with this result, and replaced the claim about all 13 agents with the number of agents whose drop is significant (7/11 as buyer and 9/11 as seller in the private field). In the main runs, the citation drop stays significant for 11 of the 13 agents once whole items are resampled, since each item recurs under the four mappings (Appendix I.5).

Adding the field lowered seller citation by 12.2 points, so we measure perception in the runs with the field and compare citation by role in the runs without it.

We revised the abstract, §1, the Detection paragraph of §7, and Appendix I.2.

AuthorsOpenReview ↗

Rebuttal by Authors

W2 / Q2 (1)

The Valuation task and the Subtle-Misaligned regime are ill-posed because the "ground-truth" price impact is unobservable from pixels. ... How is an agent supposed to infer the ground-truth dollar deduction from pixels alone

Thank you for raising this question about what the Valuation task measures. We agree that the dollar deduction cannot be read from pixels, and the submitted metric did compare concessions with dollar amounts. The task we now score asks the agent to move the price according to the type of the defect and its tier. The same dent is worth different amounts under mappings A to D, and the agent does not know the mapping. The tier order (cosmetic below structural below missing part) is shared by all four mappings, and the defect type can often be read from the photo. To make this explicit, we now score Valuation by tier order rather than by dollar amount, and we removed the phrase provably wrong. The Subtle-Misaligned photos also show small structural defects (dent, crack, rust) rather than surface marks, and 83% of the human raters who found them named a structural type (61% the exact type).

To measure directly whether agents follow the tier order, we reran each episode with the original defect-free photo. Item, role, reservation price, counterpart, and seed are all the same, and only the photo differs. We compared the agent's first offer, which comes before the counterpart can react to a citation. In the 2,970 episodes where the counterpart opened, its first offer was also identical across the two runs.

With the defect photo, the first offer drops by 1.8% of listing price [1.5, 2.1], and this holds for all 11 agents. The size of the drop, however, follows how visible the defect is, not how severe it is. It is 2.4 to 2.9% in the Obvious cells and 0.6 to 0.7% in the Subtle cells, while cosmetic (1.8%) and structural (1.9%) defects are discounted almost alike. No agent shows a significant tier order.

Adding one sentence that names the defect type (oracle) produces the order. Agents then discount structural defects 1.0 points [0.4, 1.6] more than cosmetic ones, significant for 6 of 11 agents, while the total discount stays about the same (1.8% and 1.9%). Naming the type shifts the discount toward structural defects without making it larger, so the ordering failure lies mainly in reading the defect type from the photo.

First-offer drop (% of listing price)CosmeticStructuralStructural minus cosmetic (points)Agents with a significant order
Defect photo only1.81.90.1 [-0.6, 0.7]0/11
Defect type also named (oracle)1.32.31.0 [0.4, 1.6]6/11

Human raters behave the same way. Overall their discounts do not separate cosmetic (34.1%) from structural (34.3%) defects and follow visibility instead (38 to 48% in the Obvious cells, 13 to 19% in the Subtle cells). The order appears among raters who read the type correctly, and they discounted structural defects in the Subtle cells 12.4 points [5.8, 18.2] more. Raters read the type correctly for about two thirds of the defects they found, so the type, and with it the tier, can often be read from the photo, but reading it is hard for humans and agents alike. We believe this makes the task hard rather than ill-posed. The photo swap tests whether the price moves by tier, and the oracle attributes the missing order mainly to recognizing the type.

We revised §2, §3.5, the Valuation paragraph of §7 with Figures 6B and 6C, and Appendices I.3 and I.4.

AuthorsOpenReview ↗

Rebuttal by Authors

W2 / Q2 (2)

how do you reconcile the claim that Valuation succeeds in Aligned quadrants (Figure 6B) with Table 14 showing pervasive under-adjustment across both Aligned and Misaligned turns?

Thank you for pointing out this tension. The two results diverged because the tolerance of Figure 6B in the submission (0.05R) is wide relative to defect values, so it missed the widespread under-adjustment shown in Table 14 (Table 13 in the revision). A tolerance of 0.05R exceeds the value of most defects, so 60 to 97% of turns in each cell would count as hits with no price move at all. The submission's §7 noted this, together with the fact that 77% of turns under-adjust even in the Aligned cells, and therefore read the Subtle-Misaligned result as a relative effect. You are right that the summary sentence in the abstract left out this caveat, and we have corrected it.

We have now re-examined that relative effect as well. The concession at a citing turn is about the same size in all four cells, 6.7 to 8.6% of listing price, while the defect value ranges from 10% in Subtle-Aligned to 30% in Subtle-Misaligned. Subtle-Aligned, whose defect value is close to the usual concession, looked well calibrated, and Subtle-Misaligned, whose defects are worth more, looked under-adjusted. Subtle-Misaligned holds only structural defects, which carry large values, and among structural defects it under-adjusts no more often than the other cells (+0.2 points [-6.6, 6.0]). We therefore replaced the reading that valuation fails only in Subtle-Misaligned with the conclusion of Table 14. Under-adjustment occurs in all four cells.

In the table, OA, OM, SA, and SM denote Obvious-Aligned, Obvious-Misaligned, Subtle-Aligned, and Subtle-Misaligned.

Main-run logs (13 agents, four mappings)OAOMSASM
Hit-rate at 0.05R (Figure 6B of the submission)93.1%91.7%98.8%72.6%
Turns that are hits with no price move82.4%84.3%97.2%60.2%
Defect value (median, % of listing price)25%15%10%30%
Concession (median, % of listing price)7.9%7.7%8.6%6.7%
Under-adjusted turns (concession below 2/3 of defect value)87.0%59.9%30.6%91.6%
Under-adjusted turns, structural defects only89.8%93.7%75.0% (16 turns)91.6%

The tolerance sweep and the concession analysis are in Appendix I.5.

The photo swap gives a separate measure. When only the photo changes, agents lower the price according to how visible the defect is rather than how severe it is, and so do human raters (see our answer to W2 / Q2 (1)).

We revised the abstract, §3.2, §7 with Figure 6, the ethics statement, and Appendix I.5. Table 4 of the submission moved to Appendix I.5 as Table 23.

Reviewer_aFekOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Evaluating outcomes of vision-language agents score the final result. This leaves it unclear whether a failure occurred during visual detection or price valuation. To resolve this ambiguity, this paper introduces a benchmark that adds a visual channel displaying item defects with known ground-truth price impacts. By mapping these defects onto a visibility and alignment grid, the environment makes an agent's response verifiable and isolates exactly which stage caused a mistake. The proposed benchmark is an addition over TERMS-BENCH thus making the method more general than this particular instance.

Critical strengths
  • Good topic. Diagnostics of AI behavior (especially multi step agentic ones) is crucial.
  • End to end result: focus on a specific topic of importance, take an existing benchmark and embellish it with diagnostic features, evaluate frontier LLMs on their performance, summarize key findings.
  • Easy to read, full documentation, open source software release.
Critical weaknesses
  • This is a nice paper, but not really "research". Good engineering work, something AI itself could do really well, not sure what value it adds to a conference like ICLR which is already bulging with way too high a subscription (papers submitted, reviewed, accepted/rejected).
  • This kind of diagnostic method is most useful if embedded inside an existing production grade AI deployment in the real world. I propose the authors work to get their diagnostic methods implemented and evaluated in production grade implementations and collect real world traces and showcase improvements therein.
Questions for authors

see "weakness" section above.

2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for your review and for considering the diagnosis of agent behavior an important topic.

W1

This is a nice paper, but not really "research". Good engineering work, something AI itself could do really well, not sure what value it adds to a conference like ICLR

We believe the contribution of this paper lies not in the negotiation environment, which builds on TERMS-Bench, but in a method, built on top of it, for measuring stages of evidence use that outcome scores do not reveal. Because the experimenter controls how hard the evidence is to see and how much it should move the price, the method can separate perception, disclosure, and valuation. With this method we showed three things.

First, what agents see and what they say differ. Perception, recorded in a private field that never reaches the counterpart, differs between buyer and seller by only 3.5 points, while public citation differs by 23.6 points, and this gap ranges from 3 to 56 points across agents. Disclosure is a behavior separate from perception that varies across agents, and a single outcome score cannot show it.

Second, the method locates a failure by intervening on one stage at a time. In the photo swap, no agent ordered its discounts by severity. Naming the defect type in one added sentence was enough for 6 of the 11 agents to do so. The failure therefore lies mainly in reading the type from the photo.

Third, agents miss evidence that is hard to see even when nothing is gained by hiding it. In the private field, buyers perceive Subtle defects 16.9 points less often than Obvious ones, and sellers 18.7 points less often. The drop is significant in 7 of the 11 agents as buyers and in 9 as sellers.

None of this shows in the outcome score. The agent with the highest SE+ is no exception. Claude Opus 4.6 (0.593) sees Subtle defects about as often as the other agents, yet as a seller it cites few of them. GPT-4o-mini has the highest citation precision (0.670) and the lowest SE+ (0.043).

We release the benchmark, the 324-item dataset, and per-episode traces, so the same diagnosis can be applied to other agents. Since the contribution is this reusable diagnostic and its data, we submitted the paper to the datasets and benchmarks area.

We revised the contribution paragraph of §1, §7, and Appendices I.2 and I.3.

W2

This kind of diagnostic method is most useful if embedded inside an existing production grade AI deployment in the real world. I propose the authors work to get their diagnostic methods implemented and evaluated in production grade implementations and collect real world traces and showcase improvements therein.

We agree that the method matters most when used in real deployments. Separating failures by stage, however, requires ground truth. One needs to know what evidence was present and how much it should have moved the price, and real service logs do not record this. Without ground truth one can tell that an agent mispriced, but not whether it missed the defect, saw it and stayed silent, or saw it and valued it wrongly. We therefore validated the measurement first in an environment where the ground truth is controlled.

To narrow the gap to deployment, we photographed 14 used items with real defects. We took a close-up and a far shot of the same defect, giving 14 pairs and 28 photos, and ran the same 11 agents with the private field in both roles over three seeds (1,848 episodes). The perception drop is Obvious minus Subtle for synthetic photos and close-up minus far shot for real photos, and the cited share comes from the runs with the field.

Private fieldSynthetic defect photosReal defect photos
Perception drop, buyer16.9 points7.4 points
Perception drop, seller18.7 points8.7 points
Seen defects that are cited, buyer99.1%99.6%
Seen defects that are cited, seller43.5%48.3%

Sellers withheld seen defects on real photos at a rate similar to that on synthetic ones. Perception also fell on far shots (in 10 of 14 pairs, with the reverse in one), but by less than on synthetic photos, because even far shots of real defects were seen in 84.8% of episodes and were easier than synthetic Subtle photos. The drop concentrated in Gemma-4-31B (45.2 points) and GPT-4o-mini (21.4 points), the two agents with the largest drops on synthetic photos. This experiment did not measure valuation.

The private field, the paired comparison that swaps only the photo, and the oracle that supplies the true evidence are not specific to negotiation, so they can be applied to other agents that act on photos, such as damage assessment or second-hand shopping assistants. We describe this in §8. Showing improvements on traces from a deployed service, as you propose, is left to future work.

We revised §8 and Appendix I.6.

We hope this clarifies the research contribution. We are happy to answer any further questions.

AuthorsOpenReview ↗

Kind reminder – Follow-up on our response

Dear Reviewer aFek,

As the rebuttal period is drawing to a close, we wanted to gently follow up on our response. Thank you again for your review and for recognizing the importance of diagnosing agent behavior.

We have posted a detailed response addressing both of your points. In brief:

  • On the research contribution: The main contribution is not the negotiation environment itself, but a reusable diagnostic method that separates stages of evidence use (perception, disclosure, and valuation) that outcome scores alone cannot reveal. With it, we show three findings invisible from final scores: (1) what agents perceive and what they publicly cite diverge substantially (a 3.5-point vs. 23.6-point buyer–seller gap); (2) intervening on a single stage localizes the failure: naming the defect type in one added sentence led 6 of 11 agents to order discounts by severity, which none did from the photo alone; and (3) agents consistently miss hard-to-see evidence even when there is no incentive to hide it. Since the contribution is this diagnostic and its data, we submitted to the Datasets & Benchmarks area.
  • On real-world deployment: We agree this is the ultimate goal. However, separating failure stages requires ground truth (what evidence was present and how much it should move the price), which real service logs do not record. We therefore validated the method in a controlled setting first. To narrow the gap, we additionally ran the same protocol on 14 real used items with real defects (close-up vs. far-shot pairs, 1,848 episodes) and observed qualitatively similar patterns: sellers withhold seen defects at a comparable rate, and perception drops on harder views, concentrated in the same agents that struggled most on synthetic Subtle defects.

The corresponding revisions are in §1, §7, §8, and Appendices I.2, I.3, and I.6 of the revised PDF. We would be grateful if you could take a look when convenient, and we are happy to provide any further clarification.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the three reviewers for their careful reading. Reviewer 7YLZ found the aim of separating perceptual errors from the valuation errors that follow them well motivated. Reviewer w7do pointed to the setting that couples controlled visual defects with negotiation, the defect citation interface, and the finding that outcome scores and citation behavior diverge. Reviewer aFek noted the importance of diagnosing agent behavior, and Reviewers aFek and w7do the documentation and the released traces.

All data, logs, and code are provided in the supplementary zip file uploaded with this rebuttal, including everything needed to reproduce the results reported in this rebuttal and in the revised paper.

Three questions were raised by more than one reviewer. First, does a defect citation separate what an agent saw from what it chose to say? Second, does the concession at a citing turn measure the value of that defect? Third, do the rendered defects actually satisfy their Visibility and Alignment labels?

We added experiments that measure these three things directly. Using 11 of the 13 agents in the submission, we ran 24,948 new episodes in the same setting as the main runs (mapping B, Candid counterpart, same seed), and 1,848 more on photographs of real defects. Qwen3-VL-32B is no longer served on OpenRouter, and Nemotron-3-Nano is offered there only through the free endpoint we used in the submission, which allows 1,000 requests per day, while the probes take several thousand requests per agent, so we left both out.

  • Agents record the defects they see in a field that never reaches the counterpart, which separates seeing from saying (7YLZ, w7do).
  • Each of the 270 defect items is played once as buyer and once as seller (7YLZ, w7do).
  • Each episode is rerun with the original defect-free photo, and again with the defect type named, which isolates the price response (7YLZ, w7do).
  • Ten human raters and two judge models from families other than the image generator (Claude Fable 5.2, GPT-6.1 Sol) checked the labels (7YLZ, w7do).
  • We re-examined the hit-rate metric on the main-run logs (7YLZ).
  • We photographed 14 used items with real defects and repeated the diagnosis (aFek).

We revised the paper to match the new results. We replaced the claim that the Detection drop holds for all 13 agents with the number of agents whose drop is significant, and the sentence on Claude Opus 4.6 now reports its private-field result. The Valuation conclusion no longer says that valuation fails only in Subtle-Misaligned. It now says that under-adjustment occurs in all four cells and that the price response follows how visible a defect is rather than how severe it is.

An outcome score alone cannot separate these stages, so they have to be measured separately. What agents saw and what they said diverged, and the missing severity order arose mainly at the stage of reading the defect type from the photo.

We also corrected Table 2, the image on/off ablation. While preparing the traces for release, we found that its values could not be traced to per-episode records, so we discarded them and reran the ablation twice per condition with the released code. Qwen3-VL-32B is no longer served on OpenRouter, so it could not be rerun. To keep the table reproducible from the released code and logs, we removed its row rather than keep values we could not trace, and the table now has two models, Qwen3.6-Plus and Qwen3-VL-8B. Its main-run results in Table 3 are unaffected, since their per-episode traces are released. The citation finding holds, with citation coverage falling to near zero in all four runs. The SE+ drop of Qwen3.6-Plus is larger than we reported, 0.08 to 0.11 against 0.032. The seed does not fix agent responses, so its Image values differ slightly from Table 3. The SE+ of Qwen3-VL-8B is not reproduced, so we draw no SE+ conclusion from that model. The revised §5 and Appendix G give both runs in full, and the logs are released.

Changes in the revised PDF are marked in blue, including small corrections of wording and consistency beyond the points raised, and the new experiments are collected in Appendix I. Differences below are computed before rounding, so they can differ by 0.1 from the difference of the rounded values.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.