Four months of a simulated marketplace, run by an AI, in six days

Running a marketplace is mostly operations. Orders come in, sellers ship late, parcels go missing, buyers want refunds, and sellers want to be paid on time and in full. Every one of those jobs follows rules, and most of them follow the same rules every time. So we asked a simple question: can an AI run a marketplace by itself, and how would we know if it could?

The setup: a world, a platform, an AI and a judge

We built a simulated marketplace called Kettlewick: a full home and kitchen marketplace with 12 sellers and 200 shoppers. The sellers, shoppers and carriers are simulated. Everything else is real: the marketplace runs on our own platform, the same software our customers use, on a private copy where no real money can move. And every job a marketplace operations team would traditionally do by hand is done by an AI alone.

The simulation has four parts.

• The world plays every other party. Shoppers place about 30 orders a day and write in when something goes wrong. Sellers ship, or sometimes decline. Carriers deliver, lose parcels, or send them back. Problems are planted on purpose: late sellers, lost parcels, damaged goods, wrong addresses, and buyer claims that aren’t genuine.
• The platform holds the orders, payments, refunds, payouts, disputes and the money ledger, exactly as it would for a real marketplace.
• The AI operates the marketplace, doing the work a human operations team would traditionally do. It sees only what a human operator would see: the platform, the carriers’ tracking, and its own inbox. When a case falls outside its rules, it stops, writes out what it proposes to do, and asks a person to approve or reject.
• The scorer knows the truth, such as which parcels were really lost. The AI never sees it. The scorer compares what the AI did with what actually happened, so the AI never grades its own work.

The shop moves in steps we call ticks. In each tick the world acts first, then the AI looks at the platform and does its job. One real hour is one day of shop time.

Four months in six days

Kettlewick has now been open for six real days, about four months of shop time. In that time:

• Shoppers placed 2,674 orders.
• The AI refunded 380 returns, refused 32 that came in after the 30-day window, and made 74 weekly payouts to sellers, re-checking every amount against the orders before paying it.
• It warned five sellers whose late shipments or cancellations crossed their limits, and proposed suspending the two that crossed them twice. A person approved both.
• Of 4,360 problems the world planted, the AI missed one: a late tracking number, where the scorer and the AI were timing it from two different clocks. That has since been fixed.

The number we care about most is how often the AI needed a person. Our bar for turning a simulated marketplace into a real business is at most 2 cases in every 100 completed jobs, over 30 days, with no safety breach. Over the last 30 days of shop time, Kettlewick needed a person in 0.24 of every 100. Counting only the jobs that aren’t routine deliveries and payments, it was 1.5 in 100: still under the bar.

What the rehearsal taught us

Running a full marketplace end to end taught us things no test could, and each lesson made the system better.

• How the AI’s work scales with volume. Twice, a step turned out to re-read its whole history on every tick. The second time, the simulated buyers and sellers were asking the platform for over 1,400 order numbers a step, numbers that never change. Every step now records where its time goes, and the fix nearly halved a step’s time, from 434 seconds to 224.
• When the AI should ask a person. When a buyer named two matching items, the AI proposed the one delivered last, and twice that was the wrong one. Now every case the AI is unsure of goes to a person as a written proposal to approve or reject. That is how the two seller suspensions were decided.
• What the AI costs to run. We priced every call the AI made. Keeping its standing instructions cached for an hour costs 14% less than caching them for five minutes, a direct saving once this runs in the cloud.

What it improved in our platform

A marketplace operated end to end, even a simulated one, exercises corners no test reaches. Running Kettlewick surfaces issues in our own platform that ordinary testing misses, and each one goes through our normal release process, so every marketplace on Traide gains from what the simulation finds.

What comes next

The rehearsal on a private copy is complete. Next come read-only checks on our production platform, then simulated marketplaces on our live platform, created one batch at a time with a person’s approval, then two more pilots: an outdoor-gear shop where most buyers are other companies’ AI shopping assistants, and a building-materials shop with 25 suppliers and messy price lists.

We will keep reporting what the AI gets right, what it gets wrong, and what it teaches us about the software marketplaces run on.