E-Commerce Bench

A 365-Day Long-Horizon E-Commerce Simulation for LLM Agent Evaluation

Code & data Paper (arXiv)

E-Commerce Bench is a comprehensive benchmark that evaluates LLM agents as autonomous online store owners. Each agent independently manages store setup, supplier negotiation, inventory procurement, product pricing, order fulfillment, and returns handling — with the single objective of maximizing year-end total assets (bank balance + platform wallet + pending settlement).

The environment holds 6,886 products from real e-commerce data across 60 categories, served by 576 suppliers of which 152 are fraudulent across 5 scam types, and a fixed year-long calendar of 8 promotions and 10 market events — natural disasters and supply-chain shocks — that continually reshapes demand. Much of it stays hidden, so the agent must learn which store types pay through action alone, across up to 4,000 turns per episode.

Both sides of the market are deterministic. Customers follow a fixed demand model, and a Deterministic Negotiation Kernel fixes every price, concession and accept/reject decision while an LLM only renders those decisions as dialogue. A sampled reply can never move a price, so a difference in outcome belongs to the agent rather than to chance.

We evaluate 18 models — 8 proprietary and 10 open-weight — across 7 vendor families: Anthropic (Fable5, Opus 4.6–4.8), OpenAI (GPT 5.5, 5.6 Sol), Google (Gemini 3.1 Pro, 3.5 Flash), Qwen (3.5-Plus–3.8-Max-Preview), GLM (5.1, 5.2), Kimi (K2.6, K3) and DeepSeek (V4-Pro-Preview). Each model runs 5 independent episodes in one fixed world, 90 episodes in all, and every mean is reported with its standard deviation.

365
Days
One simulated year per episode
18
Models
8 proprietary · 10 open-weight
18
Tools
One shared e-commerce toolset
6,886
Product SKUs
60 categories, real catalog
576
Suppliers
424 good · 152 bad
¥100k
Opening stake
90 episodes, one fixed world

Loading interactive charts…

Overall Performance: Balance Over Time

The primary metric — how well each agent grows its total assets over 365 simulated days.

1 / 12 Simulation Starts
Showing the full year · hit ▶ Play the year to walk through 12 key moments · click a dot to jump ·
The 2026 calendar of eight promotions and ten market events
What the year throws at the agent. Eight promotions it can opt into by choosing a discount, and ten events it cannot opt out of. The calendar is fixed, identical in every episode, which is what lets two models be compared on the same year. Promotion names are fictional, so a model cannot price a real shopping festival from memory.

Capability Profile: No Model Dominates

The primary score and the six evaluation dimensions, read together instead of collapsed into one number.

One model per vendor family — the strongest of each by mean total assets. Each axis is min-max normalized over all 18 model means, so 1.0 is the best of the eighteen and 0 the worst. Fraud avoidance, solvency, execution and learning are sign-flipped, so on every axis a vertex further from the centre is better. The dashed grey polygon is the 18-model median. Six of the seven fall below that median on at least one axis.
Click a legend entry to isolate one profile · the shape, not the size, is the point

How It Works

Six design decisions that make E-Commerce Bench a demanding, reproducible testbed for autonomous agents.

Four-layer architecture: agent loop, tool layer, environment layer, data layer
The four layers. An agent loop carries turn-based control, context management and persistent memory; a tool layer holds the 18 e-commerce tools; under both sits a deterministic environment layer — the sales-and-economy engine beside the two-layer negotiation engine — computing over a data layer derived from a real e-commerce platform.

Agent as Owner

The agent is handed a merchant account, ¥100,000 and the 2026 calendar. It opens up to 4 stores of the 12 types on offer, researches the market, sources by negotiating, prices goods, fulfils orders and handles returns. No human guidance and no pre-scripted strategy; looking something up costs the same simulated minutes as acting, so watching everything is not an option.

Deferred Cash

Revenue flows through an escrow settlement pipeline: sale → shipment → pending settlement → platform wallet → bank. Every cost hits the bank the day it is incurred, and the agent must actively withdraw to stay solvent — so a profitable plan can still run out of cash. Ten of the 90 episodes ended in bankruptcy.

576 Suppliers (152 Fraudulent)

Each supplier serves one category. 152 bad suppliers employ 5 fraud types: VIP fee extortion, quantity bait-and-switch, quality downgrade, fake urgency, and future discount traps. Detecting fraud means spending less on bad suppliers.

Information Asymmetry

Agents see product listings but never a supplier's private cost floor or fraud status. Every price, concession and accept/reject decision comes from a Deterministic Negotiation Kernel, which extends the negotiation-partner design of TERMS-Bench to repeated, cross-supplier bargaining. A separate LLM only renders those fixed decisions as dialogue, so bargaining stays in natural language while sampling can never move a price.

Seasonal Economy

Demand combines price elasticity, weekday, seasonality, 8 scheduled promotions (up to 3.5×), 10 market events and store reputation. Its only randomness is seeded from the episode, so runs reproduce exactly. Agents that fail to stock ahead of a peak miss the revenue window, and money leaves the bank the day an order is signed but returns weeks later.

Seven Readings, Not One

The primary score is end-of-year total assets (bank + platform wallet + pending settlement). Six dimensions are read beside it: negotiation quality, fraud avoidance, cash flow and solvency, operational efficiency, operations execution and learning over the horizon. No single model leads all seven.

A Day in the Life

Follow an agent through a typical business day — from market research to cash withdrawal.

Real tool calls extracted from a Qwen3.8-Max-Preview trajectory. Each step shows the actual arguments the agent passed and the result it received. This is the agent’s complete operational loop — the same 18 tools are available to all 18 models.
One turn of the agent loop, agent side against environment side
One turn. The model emits reasoning and a batch of tool calls; the batch runs in order, each call charging simulated minutes against a 600-minute day. Once the clock passes 18:00 the whole day posts at once and the day's news is appended to the response. Nothing is evicted from the transcript until it passes 120,000 tokens.

Negotiation in Action

Side-by-side comparison of how different models negotiate with suppliers — same task, different strategies.

Real transcripts from actual simulation runs. Each model initiates contact with a supplier, requests a catalog, and negotiates prices. Notice the differences in tone, strategy, and use of the negotiate JSON protocol.

Negotiation Ability

Assessing how effectively each agent bargains with suppliers to secure lower purchase prices.

What this measures: Surplus Efficiency (SE+) across 5 supplier personality families quantifies how much of the negotiable price range the agent captures. The learning curves show whether agents improve their bargaining strategy through repeated interactions with the same supplier.

Negotiation Efficiency Radar (by Supplier Family)

Average SE+ against each of the 5 honest supplier personalities (higher = better at bargaining); the adversarial family is excluded. Four models are shown by default — the top earner, the best and weakest negotiators, and Qwen3.8-Max-Preview. Click any legend entry to add or remove a model.

One negotiation session converging, and repeat deals with the anchor lost versus held
Deal anatomy, and what happens over a year. (a) One session: the supplier walks its quote down from ¥131.44 while the agent walks up from ¥102, and the supplier accepts at ¥116.74 — both sides converge, neither reaches the other’s limit. (b) and (c) are the same pair bought again and again: in (b) the agent loses its own best price and drifts upward, in (c) it holds it and keeps closing near the floor. That difference is what AnchorRatio scores.

Price Learning Curves

Each chart tracks the deal price the agent negotiates for the same SKU with the same supplier across successive rounds. The ● solid line shows Qwen3.8-Max-Preview discovering lower prices over time, while the ◇ dashed gray line shows a weaker model that never works the price down. ★ Orange step-line = best price so far. Green dashed = reference price, orange dashed = cost floor.

Fraud Detection

Evaluating each agent's ability to identify and avoid fraudulent suppliers.

What this measures: The share of total procurement spend directed to the 152 bad suppliers. Lower is better — an agent that recognizes scams spends less on fraudulent suppliers. The breakdown by fraud type reveals which scam patterns each model is most vulnerable to.

Case Study: Spotting a Fake-Urgency Scam

A real encounter between an agent and a fraudulent supplier, with the tell-tale phrases marked in place.

Bad Supplier Spend Share Lower is better

Spend on 152 bad suppliers / total procurement spend. Design principle: detecting fraud = spending less on bad suppliers.

Tool Usage

Analyzing how agents utilize the 18 available tools to operate their businesses.

What this measures: The frequency and variety of tool calls across all 18 available actions. Effective agents use the right tools at the right time — frequent chatbox usage indicates active negotiation, while balanced use of inventory, pricing, and financial tools suggests operational competence.

Tool Usage

Click a tool to view: left = per-model average call count (Qwen3.8-Max-Preview highlighted), right = real call example (from Qwen3.8-Max-Preview trajectory).

Negotiation Leaderboard

Comprehensive ranking across all negotiation metrics.

What this measures: CSE+ = the share of the bargaining range the buyer keeps on the honest deals it closes. %Oracle = realized surplus against closing every honest deal at the supplier's floor. Rounds to Deal = message turns per closed deal. AnchorRatio = overpayment on repeat orders against a reshuffling of the agent's own past quotes, so lower is better and 1.0 is chance.

Negotiation Metrics Leaderboard

Click column headers to sort. All values are 5-run averages.

Qwen3.8-Max-Preview row highlighted.

Returns & Inventory Management

How well agents manage product quality, shipping speed, and warehouse inventory levels.

What this measures: Return rate decomposes into natural (unavoidable) and controllable (caused by pricing, shipping speed, or quality decisions). Lower controllable returns indicate better sourcing and operational decisions.

Return Rate & Refund Loss

Actual return rate (bars) and refund losses (¥, dots).

Financial Discipline

Measuring financial stability, risk management, and operational consistency.

What this measures: Peak drawdown reveals the worst balance decline from peak to trough — lower means more resilient. Bankruptcy counts (triggered by 10 consecutive days of negative bank balance) indicate fundamental financial mismanagement.
Three-account structure and the cash cycle of a single order
Why a profitable plan still runs out of cash. (a) Revenue lands in escrow, settles to the platform wallet nine days after shipment, and only reaches the bank when the agent explicitly withdraws, while every cost leaves the bank the day it falls due. (b) One real order: ¥2,447 out on day 0, first cash back on day 16 — working capital locked for sixteen days.

Max Drawdown Lower is better

Largest peak-to-trough fall in total assets, as a share of the episode’s own peak. Paper Table 2.

Bankrupt Runs (out of 5)

Runs where 10 consecutive days of negative balance triggered bankruptcy. 0 = financially stable across all runs.

Learning over the Horizon

The one thing a year-long task exists to expose — and the dimension the field is weakest at.

When an agent re-orders the same item from the same supplier, does it pay less than last time? AnchorRatio answers that against the fairest possible baseline: the agent’s own past quotes for that pair, reshuffled. 1.0 means the order the prices arrived in cost exactly what a random order of them would — no learning, no anchoring. Below 1.0 the agent got cheaper over the year; above 1.0 it held an early anchor and paid more than chance. Measured over 8,647 re-orders across 2,230 repeat (supplier, SKU) pairs in the 90 episodes.

AnchorRatio by Model Lower is better · 1.0 = no better than chance

Bars are coloured by significance against each model’s own permutation null (|z| > 1.96, two-sided 5%): green beat its own shuffle, red lost to it, grey is indistinguishable.

← cheaper on re-orders than its own reshuffled quotes paid more than a random ordering would →
Read the direction, not just the absence. Fifteen of the eighteen miss their own permutation null by more than two standard deviations in the expensive direction — that is not a failure to learn, it is a systematic unrevised anchor on an early price. Only Qwen3.8-Max-Preview beats its own shuffled ordering significantly, and it is also the only model holding a positive per-day surplus trend. Two mechanisms the benchmark does not separate: context eviction destroys the tool result that carried a price already won, and the memory store that survives eviction is barely used.