acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Order Quantity: Benchmarking Behavioral Biases of LLM Inventory Agents under Uncertainty

Abstract

Large language models (LLMs) can solve quantitative problems, but inventory agents must act under uncertain demand, lead times, and partner behavior repeatedly. Existing evaluations, however, rarely distinguish limited task competence from systematic behavioral bias or move beyond a single inventory setting. We introduce AIM-Bench, a benchmark that evaluates LLM inventory agents across five environments, from a single-period newsvendor problem to multi-period and multi-echelon supply chains. AIM-Bench varies the source and structure of uncertainty and evaluates agents at three levels: operational outcomes, distance from an ex-post optimal action, and behavioral signatures such as mean anchoring, demand chasing, and the bullwhip effect. Experiments with eight state-of-the-art LLMs reveal that strong reasoning ability does not ensure reliable replenishment, because even the strongest models retain systematic behavioral biases: some models still anchor orders toward mean demand, and all evaluated models amplify demand variability in the Beer Game. These biases can remain hidden when models are compared only by aggregate cost or stockout rate. We further test two lightweight interventions. Cognitive-reflection prompting reduces anchoring for several models, information sharing substantially weakens the bullwhip effect of some models. None of these remedies is uniformly effective. Our results show that LLM inventory agents should be evaluated not only by what outcomes they achieve, but also by how their decisions respond to uncertainty.

Primary area:
datasets and benchmarks
Keywords:
large language models, benchmark, supply chain management, intelligent decision making

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 2 reviews, above 0% of the others.
Reviewer_8dAuOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces AIM-Bench, an LLM agent benchmark of inventory management tasks involving various types of uncertainty. Agents must manage uncertain demand, lead-time, and partner-behaviors to perform well on the tasks. The work evaluates LLM behavioral biases and interesting inventory dynamics that can arise from imperfect decision making under uncertainty.

Critical strengths
  • AI decision making under uncertainty is both interesting and has broad real-world implications.

  • From the descriptions available, the benchmark design appears to cover a variety of scenarios and places agents under varying sources of uncertainty.

Critical weaknesses
  • The AIM-bench benchmark itself is not provided in a supplemental file nor explained in sufficient detail to be reproduced. Some of the tasks are only explained at a cursory level. The paper presents itself as introducing a benchmark, presumably for others to use, so this is a critical omission.

  • Only a single trajectory was run for most model-condition pairs, so we don’t have information on the variance in the results.

  • I find it difficult to determine from the figures which agents perform well and which perform poorly. For example, in Figure 4 shows “inventory ratio”, but given the insufficient explanation of the tasks, I don’t know whether a high inventory ratio is good or bad. Better explanations in figure captions would help, as would including random baselines or perfect baselines where available.

  • Baselines are not provided. The paper mentions that simple algorithms may be able to perform some or all of the inventory management tasks they evaluate (this claim is hard to verify due to the minimal information given about the actual tasks), but does not compare LLM performance to those simple algorithms.

  • The paper would benefit from an expanded discussion, discussing what the results imply for real inventory management and whether the results have broader implications beyond inventory management.

Questions for authors
  • What precisely are the tasks in the benchmark?
  • Why was only one trajectory run per model-condition pair?
  • Is there a reason someone would use an LLM on the benchmark tasks, rather than a simpler algorithm?
Reviewer_i27gOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents AIM-Bench, which tests LLM agents on inventory decisions in five settings: newsvendor (NVP), multi-period replenishment (MPR), Beer Game (BG), two-level warehouse networks (TWN), and dual-sourcing supply chains (SCN). Performance is measured by cost, inventory levels, stockouts, how far orders deviate from hindsight-optimal decisions, and behaviors such as mean anchoring, demand chasing, framing sensitivity, and demand amplification.

The study compares eight 2026 LLMs and tests two interventions: cognitive-reflection prompting and information sharing. Seven of the eight models always choose the optimal newsvendor quantity (q∗=120q^*=120), but all show demand amplification in the Beer Game. Information sharing helps some models but hurts others, while action distance distinguishes MPR policies that achieve similar costs.

Critical strengths
  1. Evaluation beyond final outcomes. The paper uses inventory theory to assess not just costs but also decision quality and behavior, drawing on optimal order quantities, ex-post replenishment decisions, and order variability. The five environments cover different sources of uncertainty, including demand, lead times, and supplier behavior. The finding that seven models perform equally well in NVP but differ substantially in MPR shows that solving a static problem does not guarantee good dynamic decisions.

  2. Clear reporting of limitations. The paper shows that frontier models no longer exhibit the typical human pull-to-center bias in the standard newsvendor task. It also acknowledges the limitation of evaluating each model on a single trajectory. These findings are worth reporting, even if they weaken some of the paper's broader claims.

Critical weaknesses
  1. Several main claims are weakly supported or inconsistent with the results. The paper claims that some models still anchor toward mean demand and that cognitive reflection reduces this bias. However, seven of eight models choose exactly q∗=120q^* = 120 in every round under both frames. The anchoring evidence therefore comes from just one model (DeepSeek-V4-Pro) and 20 decisions. For models with constant orders, ρ(qt,dt−1)\rho(q_t, d_{t-1}) is undefined, not zero. DeepSeek-V4-Pro's correlation of ρ=0.327\rho = 0.327 across 19 pairs is not significant (p≈0.17p \approx 0.17), making the reported reduction to 0.192 unconvincing.

    The paper only tests the high-margin case (c=3c = 3), despite stronger human anchoring often appearing in low-margin settings. Its setup (c=3c = 3, r=10r = 10, U[50,150]U[50, 150]) is also a textbook example, so choosing q∗q^* may reflect memorization rather than genuine decision-making ability. Beer Game results, including universal demand amplification and mixed effects of information sharing (e.g., Grok-4.6: 1.97 to 2.13), rely on just one trajectory per condition.

    There are also numerical inconsistencies. Grok-4.6's MPR cost is reported as 224 in the generational comparison but about 89 in Finding 5 and Appendix D. Other discrepancies include DeepSeek-V4-Pro's cost (134.65 vs. 129.22), Kimi-K3's cost (136.83 vs. 143.61), Grok-4.6's stockout rate (0.117 vs. 0.07), and Gemini-3.7-Flash's cost/IR/SR (236.79/2.07/0.383 vs. 232.35/2.36/0.29). Results for previous-generation models are also missing.

  2. Missing rational baselines make the behavioral metrics hard to interpret. The paper provides no classical policies, such as base-stock or (s,S)(s,S) for MPR and BG, or learned baselines such as MABIM. This is especially important for the bullwhip results. With a fixed ordering cost (FC=5FC = 5), even an optimal (s,S)(s,S) policy may batch orders and amplify demand variability. Thus, β>1\beta > 1 does not necessarily indicate irrational behavior. The paper acknowledges this possibility but provides no rational-policy comparison. The bullwhip definition also changes between Eq. (11), which compares adjacent echelons, and Finding 4, which reports L1/L0L_1/L_0 and L2/L0L_2/L_0 relative to customer demand.

    The MPR action-distance metric has a similar issue: it compares decisions with clairvoyant ex-post orders, mixing unavoidable forecasting errors with poor decisions. Without a realistic reference policy or evidence that distance predicts out-of-sample cost, its added value remains unclear.

    Finally, two of the five environments receive little analysis. In TWN, seven of eight models have a stockout rate of 1.00, despite inventory ratios ranging from 0.0035 to 13.37, suggesting a possible feasibility or interface issue. Neither TWN nor SCN appears in the main findings, making the paper's broader claims across five environments difficult to support.

Questions for authors
  1. Please reconcile the MPR numbers between Section 5.2 and Appendix D, and explain the Grok-4.6 MPR cost of 224 in the generational comparison. Can you report the previous-generation results in full?
  2. Which models, other than DeepSeek-V4-Pro, show nonzero anchoring in any condition, as the abstract's "some models" and "several models" imply? What are the results for the low-margin NVP condition, and how is ρ\rho handled when orders have zero variance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.