- Smaller models now do most of the helper work: helper jobs run on our large model fell from 65% to 14%.
- 45% less for the same work: priced on the large model, the same helper work would have cost 1.8 times as much. In dollars, that is about 3% of our whole AI bill.
- No drop in quality we could find: in the 131 smaller-model jobs whose follow-up we could read, our lead AI recorded no corrections. Most results were used without a written check.
- Faster writing: on the same kind of job, the mid-size model wrote 37% faster than the large one, and the small model about twice as fast.
- The bigger bill is elsewhere: helpers are 4% of our AI spend. The other 96% is the lead AI, and most of that is it re-reading what it already knows.
Key takeaways
Can a smaller, cheaper AI model do routine work without the work getting worse? We measured it on our own work for nearly four weeks. Most of the AI work behind Traide is done by helpers: narrower jobs that a lead AI hands off, like finding something in our code, reading a web page or reviewing a change. Since 23 October, a router has chosen the model size for most of those jobs. The short version: the helper work cost 45% less than it would have on our large model, we found no drop in its quality, and helpers turned out to be a small part of our AI bill.
What we changed
We build Traide with a crew of AI models of different sizes, as we described in how we build with AI. One lead model runs each working session. It plans, decides and checks, and it hands narrower jobs to helpers. A helper can run on a small, fast model; a mid-size model; or the same large model as the lead.
Our published plan sent search and research to the small model. When we measured it in October, practice did not match. A helper took on the lead’s model unless told otherwise, so our search helper ran on the large model every time, and so did 65% of all helper jobs.
So on 23 October we switched on a router: a small program that reads each helper job before it starts and picks the smallest size our rules allow. Searching and reading go to the small model, building and reviewing to the mid-size one, and design and hard debugging to the large one. One rule overrides the rest: anything that touches money never runs below the mid-size model. Some helpers have their size fixed in advance instead, and the lead can still name a size itself.
How we measured it
Every AI request leaves a record: which model answered, how much it read and wrote, and how long it took. We compared the 26 days after the switch with the 33 days before it, and priced every request at today’s public list prices. To judge quality, separate AI reviewers read the lead’s notes on each job without knowing which model had done the work.
Smaller models now do most of the helper work
Before the router, 65% of helper jobs ran on the large model. Since, 14% have. The mid-size model now does 74% of helper jobs, up from 28%, and the small model 11%, up from 2 jobs in 81. Every job that was given a size ran on exactly that size.
Cost: 45% less for the same work
Priced as if the large model had done it, the same helper work would have cost 1.8 times as much, so it cost 45% less. About three-fifths of that saving came from the router’s own choices. The rest came from sizes fixed in advance or named by the lead. Before the router, the same comparison showed almost no saving at all.
Code review shows it most plainly. Reviews on the mid-size model cost a median of 67 cents and took 2.7 minutes. Over two days in October, 15 reviews were started in a way the router does not see, and they ran on the large model or bigger. Their median was $1.51 and 3.2 minutes, for reviews of about the same size.
In dollars, the saving is modest: about 3% of our whole AI bill for the period. The reason is below.
Quality: no drop we could find
Our lead model is required to check what a helper brings back before relying on it, and it writes notes as it goes. We had separate AI reviewers read the lead’s notes after 188 helper jobs, with the model size hidden, and sort each one: an error found, a correction, checked and found right, used as given, or unclear.
Since the switch, the small and mid-size models did 158 helper jobs, and we could read the lead’s follow-up for 131 of them. It recorded no errors and no corrections. In 18 of them it re-checked the work and found it right. The one correction it noted since the switch was to work by the large model. The only claims we saw it reject came from a batch of research reports the large model wrote before the router existed.
That is not proof the smaller models never slipped. Most results were used without a written check, and 21 of the 131 notes were unclear. But by the same measure, the large model did no better.
We also checked the router’s choices. A reviewer read 93 jobs the router had sized, without seeing what it picked, and chose the size our rules call for. They agreed 76% of the time. In every disagreement, the router had picked a bigger model than needed, never a smaller one. When it is wrong, it costs money, not quality.
Speed: smaller models write faster
On the same kind of job, the mid-size model wrote 37% faster than the large one, and the small model about twice as fast. The wait for each step shrank less: a median of 5.1 seconds on the mid-size model and 3.9 on the small one, against 5.5 on the large one. The routing decision itself takes about three hundredths of a second.
What surprised us
- The helpers were never the big bill. Even before routing, helper work was only 12% of our AI spend. Now it is 4%, partly because the lead’s own work grew. The other 96% is the lead model, and most of that cost is the lead re-reading its own earlier work: about 99% of the text it takes in is context it has already seen. The next saving is not a smaller model. It is a lead that re-reads less.
- Two of our own safeguards tripped over each other. Since 3 November, the check that approves each AI action has been refusing helper jobs that the router adjusted after the check had looked at them. 37 of 66 jobs were refused at first and sent again, each time on the size the router had chosen. Nothing ran on the wrong model, but each retry took about 15 seconds.
- Some work went around the router. The 15 code reviews above were started in a way the router does not see. They cost more than twice what the mid-size model would have.
- The money rule is too cautious. Of the 93 jobs we checked, it flagged 53 as touching money, and 19 really did. It missed none. Most of its false alarms were web research that mentioned money without touching ours: 13 such jobs went to a bigger model than they needed. Under the narrower rule we have now decided on, 44 of the 93 belong on the small model. The router sent 17.
- Our own cost report was wrong. It priced part of the mid-size model’s work at twice the real rate, which made routing look less useful than it is. We caught it while checking these numbers.
What comes next
Three fixes, each measured before and after in the same way. We will stop the approval check and the router tripping over each other. We will give the reviews that go around the router a mid-size default. And we will narrow the money rule, so web research can use the small model while anything that reads the code that handles payments keeps the mid-size one. Then the larger saving: cutting how much the lead model re-reads.
What it means for you
If you run AI on real work, three lessons from these four weeks:
- Measure what ran, not what was planned. For two weeks, our published plan said search went to the small model, while our search helper ran on the large one every time.
- Put smaller models where mistakes can be caught, and measure the catching. Our lead is required to check helper work, but its notes show an explicit check on only about one job in seven, so we cannot yet say how much safety that buys.
- Look at the whole bill. Routing was worth doing, but most of the cost was somewhere else.

