Over the last couple of years I have sat with more than a hundred companies who built something with AI — dashboards, products, tools. Founders on a mission, screens full of charts, metrics moving in real time, an assistant answering questions right there in the interface. Polished. Confident. Fundable-looking.
I ask one question. It has never taken more than one.
"The data you're presenting — is it seeded, fake data, or is it actually real?"
Here is the part I want you to sit with: I usually don't even have to ask the founder. I ask the product's own AI assistant — the one proudly embedded in the demo. And every time, the AI admits it. Seeded. Synthetic. Generated for demonstration purposes. The beautiful dashboard is a costume, and the intelligence inside it will cheerfully tell you so, because the model has no loyalty to the pitch.
Every time. Out of more than a hundred reviews, I am still waiting to lose count of the exceptions, because I can count them without using my hands.
One scene, a hundred meetings
Let me collapse them into one scene — a composite, details blurred, pattern exact, because after this many it plays like a rerun. A founder shares a screen. The dashboard is genuinely beautiful: revenue by region, technician utilization curves, a churn-risk panel with little red flags, an assistant docked on the right that answers questions about the numbers in full, confident sentences. The founder says the words real time at least twice. Somebody on the call types "this is incredible" into the chat.
I type my question into the product's own assistant: "Is the data on this dashboard connected to live systems, or is it sample data seeded for this demo?"
Three dots. Then, politely, perfectly formatted: "The data shown here is sample data generated for demonstration purposes."
There is always a pause after that. I have learned not to fill it.
The four deflections
What happens next is so consistent I can score it like a bingo card. After the confession, one of four sentences arrives:
- "We're literally about to integrate real data." The eternal two weeks. If the plumbing were nearly done, the demo would already run on it — real data is the most impressive feature a data product can have, and its absence is the roadmap.
- "That's on the roadmap." The roadmap is where the hard 80% goes to be applauded without being built. Roads are for driving; this one is a mural of a road.
- "Our pilot customers love it." Pilots run on seeded data aren't traction — they're a focus group admiring the costume. Love of the demo tells you the interface is good. It tells you nothing about the product, because the product isn't there yet.
- "You just don't get the vision." See above. I get the vision fine. Vision is the part I do for a living. What I don't get is the invoice a customer will pay for numbers nobody wired to anything.
I have a name for this whole category now: cardboard AI. It looks exactly like the appliance — same shape, same paint, same glow — and it holds no weight, because the data inside is painted on. It photographs beautifully. You just can't put anything on it.
Why this keeps happening
Because AI made the appearance of a product nearly free. A weekend of prompting gets you an interface that looks like six months of engineering. But wiring it to reality — real data flowing from real systems, cleaned, secured, permissioned, reconciled — is the actual work, and the actual work didn't get cheaper. So the demo gets built, the data gets seeded "just for now," and "just for now" quietly becomes the go-to-market plan.
And there's a second, harder truth underneath: AI often lies. Not maliciously — structurally. A language model's default failure mode is confident fabrication. Point one at an empty database and ask for insights, and it will not say "there's nothing here." It will render you a trend line. It will invent the very numbers your customer is about to make payroll decisions with. A dashboard without real data plumbing isn't an early version of a product. It's a hallucination with a user interface.
The literacy test
I'll say the unpopular part out loud: not being a developer is the worst mistake you can make in this game — not because building without engineers is forbidden, but because you can't secure what you can't name. Here's the test, and it isn't gatekeeping, it's a mirror:
- Do you know what a state machine is — and what your product does when a job is half-finished and the connection drops?
- Do you know what user-based roles and permissions are — and can you prove customer A can never see customer B's data, even when someone asks your AI nicely?
- Do you know what hardening means — and has anyone hostile ever pointed themselves at your product on purpose?
If those words are furniture to you, that's fine — genuinely. It just means you're a prototyper right now, not a product company. Those are different jobs. Both are honorable. Only one of them should be allowed near another company's operations.
Visionary is not a costume
Whenever I say some of these products shouldn't exist, somebody reaches for the visionary defense: every breakthrough looked crazy once — you of all people should know that.
I do know that. I've been called crazy for four years and written about what it costs. So hear the distinction from someone who has lived both sides of it: visionaries see things no one else sees — and then build things that work. The vision gets tested against reality, and reality signs off. Pretending to be an expert is the opposite motion: borrowing the visionary's costume to sell something reality has already voted against. One of these earns the beating and the vindication. The other just collects checks until the customer finds out.
Here's a real specimen making the rounds in the trades right now: a vendor builds you a second website — a duplicate of your main site, stripped of any design — and tells you it will help you "rank in AI search." To a non-expert it sounds plausible: more websites, more surface area, more chances to be found. Here is what actually happens. AI search and answer engines are trying to resolve one thing: who you are — one entity, one authoritative source of truth, which is the entire game of AI visibility. A duplicate site splits your identity in two, dilutes the authority you spent years earning, competes with your own content, and hands the machines two conflicting versions of you to distrust. And when a customer — or their AI agent — lands on the designless copy, the impression is worse than not being found at all. You paid to hurt yourself. The vendor got paid either way.
That is not vision. That is a product that actually damages the business it was sold to — wearing vision's language as a disguise. The test that separates them is the same one that runs this whole essay: the visionary's product survives the question. The pretender's product survives only the demo.
What AI is actually for in untrained hands
Prototyping. And I mean that as a compliment — the prototype is the most valuable artifact in business right now. It proves the idea, it recruits the co-founder, it raises the money, it teaches you what the product wants to be. Build it in a weekend. Show it to everyone.
Then do the grown-up thing: hire a real development team — people who read your prototype as a specification, not a foundation — and let them build the version with state machines, permissions, hardening, harnesses and guardrails, and data that is real because the plumbing is real. Because it looks easy doesn't mean it is — and it certainly doesn't mean it's a product you should take to market.
The reviews that pass
I want to tell you about the exceptions, because they're the point. A handful of times — few enough to remember each one — the meeting went differently. I asked the question, and the founder answered before the AI could: "Seeded — on purpose. Here's why. And here's the real integration running on my laptop against my own company's data. Meet Sarah, she built the pipeline; ask her anything." Then Sarah told me, unprompted, exactly which edge cases still scared her.
Notice what passed the review: not polish — honesty about the gap, plus proof the gap was being closed by someone qualified to close it. Knowing where the bodies are buried is the first credential. Those founders got my time, my feedback, and in some cases my introductions, because that posture is exactly how real products get born. If that's you: come back when it's real. I will be your loudest advocate — being early and honest is the whole club membership, and the club is small.
The stakes are other people's livelihoods
This is the sentence I wish I could staple to every pitch deck: the contractor who buys your tool is not buying software. She is betting her dispatch board, her pricing, her payroll, her customer relationships — the company that feeds her family and her employees' families — on the proposition that your product is what it looks like. If what it looks like is a real system, and what it is is seeded data wearing an interface, you are not "iterating in public." You are risking other companies' livelihoods with a fake AI product — cardboard AI — and the word for that isn't startup. It's negligence with a better logo.
And because I keep my predictions where people can grade them, put this one on the record: before the end of 2027, a cardboard-AI product — seeded data sold as a live system — will be at the center of a public lawsuit or regulatory action in our industry. When it happens, everyone will act surprised. Nobody who asked the question will be.
Ask the question
If you're evaluating any AI product — as a buyer, an investor, a partner — you now have my entire due-diligence secret, and it costs nothing:
"Is this data real or seeded?" Then ask to see your own data flowing through it, end to end, with your own eyes.
Ask the founder. Better yet — ask the product's own AI. It's the one participant in the demo with no incentive to impress you, and in more than a hundred meetings, it has never once lied to me about being fake.
The irony is perfect, and it's the whole lesson: the AI tells the truth about the product more reliably than the product tells the truth about the data. Build accordingly.