Before you evaluate tools, build business cases, or lead pilots, you need a clear mental model of what AI actually does. This lesson gives you a framework that cuts through the hype and lets you make sharper decisions about when AI is the right answer — and when it isn't.
Every vendor pitching you an AI tool wants you to believe it can do everything. Every breathless news article makes AI sound like it will either save civilization or end it. Neither is useful to you as a manager trying to run a team and hit quarterly goals.
What you need is not enthusiasm or anxiety — you need a working mental model. A way of thinking about what AI does that is accurate enough to make good decisions, practical enough to apply quickly, and durable enough to use as AI continues to evolve.
Here is one that holds up: AI is a pattern-completion engine. It learns patterns from enormous amounts of existing data and uses those patterns to complete new inputs. That's it. When you type a prompt into ChatGPT, you are providing the beginning of a pattern, and the AI is completing it based on what it has seen in training data.
This simple model explains both AI's capabilities and its failures — and it will help you evaluate any AI claim in about 30 seconds.
Everything you will ever be pitched by an AI vendor, everything your team will ever ask about, falls into one of four categories. Understanding these categories is the single most useful framework a non-technical manager can have.
Within those four task types, AI performs best when the following conditions are present:
Understanding AI's failure modes is as important as understanding its strengths. Here is where managers consistently get burned:
AI is pattern completion. It is not reasoning in the human sense. Ask it to analyze a situation it has not seen in training data — a truly novel business problem, an unusual regulatory situation, an unprecedented organizational challenge — and it will produce something that sounds confident but may be nonsense. It completes the pattern of "what a confident answer looks like" rather than actually reasoning through the problem.
AI hallucinates. This is not a bug that will be fixed — it is a property of how the technology works. AI will invent specific facts, fabricate citations to papers that do not exist, and state incorrect statistics with complete confidence. Any AI output that includes specific numbers, named sources, or specific dates must be independently verified.
AI only knows what you tell it. It does not know your organization's history, your team's dynamics, the political sensitivities of a particular stakeholder, or the unwritten rules that govern how decisions get made in your context. When AI gives you advice on complex organizational situations, it is working without this context — and the gaps show.
AI is very good at producing competent, conventional work quickly. It is not good at genuine originality. The insights that come from combining two things nobody has combined before, the creative leaps that change how an industry thinks, the contrarian position that turns out to be right — these require human judgment. AI can assist with execution but rarely with the generative insight.
You will be pitched a lot of AI tools. Here is a five-question filter that surfaces the most important information in ten minutes:
| Question | What a Good Answer Sounds Like | Red Flag Answer |
|---|---|---|
| What specific task does this do? | "It classifies incoming support tickets into 12 categories and routes them to the right team." | "It uses AI to transform your entire operations." (No specifics.) |
| How accurate is it? | Specific accuracy rate with test methodology. "92% accuracy on a held-out test set of 10,000 tickets." | "It's very accurate" or "it gets better over time." (No numbers.) |
| What happens when it's wrong? | Clear description of failure modes and how humans catch and correct errors. | "It rarely makes mistakes." (Evasion — all AI makes mistakes.) |
| Can you show me a live demo on our actual data? | Yes. Demo on realistic data similar to yours. | "We'll set that up in phase 2." (They know it won't perform on real data.) |
| What do customers who have been using this for a year say? | Specific reference customers you can call, with measurable outcomes they've achieved. | "We're still early in rollout." (No proven track record.) |
Think about the work your team does on a weekly basis. List 10 recurring tasks — the things that happen every week or month. Then, for each one, make two assessments:
For each task you rate "Strong candidate" or "Possible," note the one specific condition that makes it suitable — or the one concern that needs to be addressed before deployment.
Keep this list. You will use it in Day 2 to frame your tool evaluation and in Day 3 to build a business case.
Our 3-day AI bootcamp includes a full-day leadership module with live vendor evaluation workshops. Five cities. $1,490 per seat.
Reserve Your Seat →