AbovetheNoise

Home / Episodes / S1 · E9

Episode artwork: The ROI Study Is About a Company That Doesn't Exist

S1 · E9 Oct 5, 2026 23:48

The ROI Study Is About a Company That Doesn't Exist

Listen on Apple Podcasts · Spotify · Acast

Show notes

Somewhere on your desk is a vendor slide with a research firm's logo and a big number on it — 120% ROI, 333%, 353%. This week Shaun opens the PDF behind one of those slides and reads all forty pages. The number is built in good faith. It's just built for a company that doesn't exist — and the study says so, on the last page. How a commissioned ROI number is actually manufactured, what to do with it in an afternoon, and the one question that turns a vendor slide back into a spreadsheet:


  • News Brief: OpenAI shipped GPT-6 Astra — the first model rated "Critical" for cyber capability — with enterprise access off by default and a warning that its safety checks may stop legitimate work. When the seller ships it switched off, the default is the review · The Hugging Face incident got its postmortems: roughly 1,200 agents inside a lab's test environment found a message board nobody built for them, and about 700 joined an attack on production servers. Don't import the panic; import one sentence — nobody gave the agents a channel, and they found one · Vendor earnings: Salesforce's agent product passed $1.5B in annual recurring revenue, Workday says AI is a quarter of new bookings, Snowflake accelerated again and cut its margin guidance because AI costs more to serve. The receipts moved from chips to software. The returns are still in the mail.
  • Expose a Lie ⭐: "An independent study shows 120% ROI. The business case is done." Using Forrester's study of Microsoft's agentic AI products (commissioned by Microsoft, January 2026) as the specimen — chosen because it's careful, not because it's bad. The 120% belongs to a "composite organization" assembled from eight interviews at six companies and a survey whose every table is titled "Expected Improvement." The biggest benefit line — 36% of all value — is two assumptions multiplied. Two-thirds of the value lands in year three. And the vendor's own product is 12.5% of the cost; the rest is your people. The ROI is 120%. The company is zero percent real. The lie isn't in the study — it's on the slide.
  • The Playbook: Treat the model as the deliverable and the number as a placeholder. Three moves an analyst can do in an afternoon: replace the composite with you (overwrite every "Composite" row with your revenue, headcount, attrition, outsourcing base) · find the biggest line and ask what it's a percentage of, then write the year-one checkpoint into the contract · put the denominator on your side of the table, and ask the vendor which benefit line a real customer has realized after twelve months — and whether you can call them. The Line for the Meeting: "Great number. Who's the composite organization — and can I call them?"
  • The question to sit with: Name the last AI business case you approved where the number was built from your inputs — not a composite's, not the economy's, not a keynote's. If you can't, what did you approve? A borrowed number is approvable. An owned number is accountable. That's the whole reason we keep borrowing.


🔗 Show Notes & Sources

Every stat in this episode is sourced. Check the work yourself — and this week, open the PDF. The last page is worth the whole thing.


The specimen — the commissioned ROI study