Started 2w ago 4 replies 1.2K views
Our team built an internal assistant that answers questions about our policy documents. The bill came in at three times my boss's estimate, and he's convinced we're being overcharged. His model of pricing is "per question", like a phone call.
I know the cost is based on tokens, and that our prompts include long documents every time. But I'm struggling to explain it in one minute without him glazing over. How do you explain this to a non-technical manager?
Accepted answer
The analogy that works best for me: it's priced like a photocopier, not a phone call. You pay for every page that goes through the machine, both the pages you feed in and the pages it prints out. A short question stapled to a 40-page policy document is a 41-page job, every single time.
Then show him one real request: how many tokens were the question, the attached documents and the answer. The documents will be 95% of it. We have a plain-English explainer you can forward: What Is a Token?
Once he gets it, the fix becomes obvious: stop sending whole documents. Retrieve only the relevant sections for each question. We cut a similar bill by about 70% that way, and answers got better because the model wasn't wading through irrelevant pages.
Show him the bill broken down by feature, not by month. 'The policy assistant cost X, of which Y was documents sent with every question' is a story. A single total is just a scary number.
The photocopier line landed instantly. He actually said 'so we're photocopying the handbook every time someone asks about holidays?' Retrieval project approved this morning.
