How to Tell If an AI Vendor's Claims Are Real or Just Marketing

Every pitch deck in 2026 says "powered by AI." Every vendor demo looks impressive. But a slick demo and a working product are two very different things, and most buyers don't find out which one they bought until months — and a lot of budget — later.

If you're evaluating an AI vendor for your business, here are four things to check before you sign anything. None of them require a technical background. They just require asking the right questions and paying attention to how the vendor answers.

1. Ask for Real Testimonials, Not Just Logos

A wall of recognizable client logos on a website tells you almost nothing. It doesn't tell you what those clients actually use, how long they've used it, or whether they're happy. A genuine testimonial does.

When you're talking to a vendor, ask for:

• A reference client you can actually speak to, not just a quote on a slide
• Specifics: what problem the AI solved, what changed before and after, what it took to get there
• How long that client has been using the product — six months of stability tells you more than a six-week pilot

If a vendor hesitates to connect you with a real client, or every testimonial is vague ("game-changing," "incredible results," no numbers attached), treat that as a signal, not a footnote. Vendors with real deployments are usually proud to put you in touch with the people running them.

2. Look for Actual Deployed Projects, Not Just Demos

A polished demo is built to demo well. It's a curated, best-case scenario, run by the people who built it, on data they chose. None of that tells you how the system behaves with your messy, real-world data, your edge cases, or your team using it daily.

Ask the vendor directly:

• "Can you show me a project that's been live in production for at least six months?"
• "What broke, and how did you fix it?"
• "How many users actually use this day to day, not how many signed up?"

A vendor with real deployed work will have an honest answer to "what went wrong," because something always does. A vendor who only talks about what could go right, or pivots every question back to the demo, is likely still validating the idea on your dime.

3. Find Out If They're Honest About Training vs. RAG

This is the one most buyers skip because it sounds technical, but it has a direct, practical impact on cost, accuracy, and how quickly the system can be updated.

"Trained on your data" and "retrieval-augmented generation (RAG) over your data" are very different things:

• Fine-tuning / training changes the underlying model itself using your data. It's expensive, slower to update, and only worth it for narrow, repeatable tasks.
• RAG keeps the base model as-is and feeds it relevant information from your documents or database at the moment of the query. It's cheaper, easier to update, and is what most "AI trained on your data" claims actually mean in practice.

There's nothing wrong with using RAG — most production AI systems today are built this way. The problem is vendors who say "trained on your data" when they mean RAG, because it makes the system sound more bespoke and defensible than it is. Ask directly: "When you say trained, do you mean the model itself was fine-tuned, or are you retrieving from a knowledge base at query time?" A vendor who can answer that clearly, without dodging, understands their own product. One who gets vague is selling you the idea of customization, not the substance of it.

4. Check If They're Transparent About Running Costs

The sticker price for an AI feature is rarely the real cost. Every query to a large language model costs money — in API fees, compute, or both — and that cost scales with usage, not with the size of your initial contract.

Before committing, ask:

• "What's the cost per query or per user at our expected volume, not your demo volume?"
• "Who absorbs the cost if usage grows faster than projected — us or you?"
• "Are there hidden costs for storage, retraining, or scaling beyond a certain number of users?"

A vendor who's confident in their economics will walk you through this without flinching, because they've already done the math. A vendor who avoids the question, or buries it in a "let's discuss as we scale" line, often hasn't worked out whether their margins survive contact with real usage — which becomes your problem the moment it becomes theirs.

The Bottom Line

None of these four checks require you to understand machine learning. They require the vendor to be specific where it's easy to be vague, and honest where it's tempting to oversell. Real AI capability holds up under specific questions. Marketing dressed up as AI capability usually doesn't.

Before your next vendor conversation, ask for one verifiable reference, one production deployment older than six months, a straight answer on training versus RAG, and real numbers on running costs. How they respond will tell you more than anything on their website.