LESSON 03
WHERE IT WILL BE
CONFIDENTLY WRONG.
Everything you read about AI capability is written by someone with a reason to emphasize the upside. This page exists to state the limits plainly, because knowing the boundaries is most of what separates useful from expensive.
Arithmetic you do not check
Numbers are the first and most expensive failure, because numbers are where wrong output stops being embarrassing and starts being costly.
A language model is not a calculator. It produces a number that looks right in context. On simple arithmetic it usually lands. On a multi-step calculation with your own figures, it will sometimes drop a line, apply a percentage to the wrong base, or carry a number forward incorrectly, and the resulting page of work will read as immaculate.
Some tools now hand arithmetic to an actual calculator behind the scenes, which genuinely helps. It does not help with the more common failure, which is that it calculated the wrong thing correctly because it misread what you asked.
The rule is short. Any number that ends up in an offer, a contract, a budget, an invoice or a tax document gets checked by a person against the source. Use it to lay out the structure of a calculation. Do not use it as the last step before you sign something.
Anything current
A model's training has an end date, and it does not experience the passage of time after that. Ask about something recent and you may get an answer drawn from a world that has moved on, delivered in the present tense.
Tools that search the web narrow this, and you should prefer them for anything time-sensitive. But the summary of what it found is still generated text. It can misread a source, blend two sources, or cite something that supports a weaker version of the claim it just made.
Current facts include more than the news. Prices. Rules and ordinances. What a property sold for. Who owns what. Which lender is doing what this quarter. What your own pipeline looks like today. That last category is not a limitation of the model so much as a reminder that it can only see what you put in front of it.
This is one piece of a bigger picture; the deal calculators and learning resources has the rest.
Legal, contract and compliance language
This is the hardest line on the page and it is worth being blunt about.
Legal language reads as highly patterned, which makes it exactly the kind of text these systems imitate well. You will get a clause that looks professional, uses the right vocabulary, and is structured the way real clauses are structured. That is the danger, not the benefit. A contract term that is ninety-five percent right is not ninety-five percent useful. The five percent is the part that decides what happens when something goes wrong.
The same applies to anything with a compliance dimension. Employment language. Disclosures. Insurance. Anything a regulator reads. And it applies to jurisdiction, which the model will cheerfully ignore, handing you a provision that is ordinary in one state and unenforceable in another.
Use it to understand a document, not to produce one. Asking it to explain a clause your attorney drafted, in plain English, so you know what you are signing, is a genuinely good use. Asking it to draft the clause is not. Contracts and compliance language come from your own attorney, and your CPA handles the tax side.
Anywhere confidently wrong carries a real cost
That phrase is the general rule and everything above is a special case of it. Before you rely on an output, ask one question: what happens if this is wrong and nobody notices?
| Cost if wrong | How to use it |
|---|---|
| Low. You reread it and fix it. | Use freely. Drafting, summarizing, rewriting, brainstorming. |
| Medium. A customer sees something sloppy. | Use with a real review pass before it leaves. |
| High. Money moves, a contract binds, a regulator reads it. | Use to prepare and understand. A qualified person produces the final version. |
Most owners never do this sorting, which is why the failures cluster. The tool is used identically for a rough email and a purchase agreement, and the review effort is identical too, which is to say close to zero.
It does not replace judgment
Judgment is the accumulated pattern recognition of someone who has been wrong before in this specific domain and remembers the shape of it. It is knowing that this seller's tone on the phone means the family is not aligned yet. It is looking at a rehab scope and feeling that the number is light, before you can articulate why.
A model can produce a well-organized analysis of a decision. It cannot carry the consequence of being wrong, and consequence is where judgment comes from. Ask it to list what you might be missing, which it is good at. Do not ask it to make the call.
There is a subtler version of this failure. Used constantly for thinking, it erodes the thing it was supposed to support. If every decision arrives pre-framed by a model, you stop practicing framing decisions, and that skill atrophies quietly enough that you will not notice until it matters.
It does not replace relationships
In a business where people are handing you a house, a job, or a financial situation they are embarrassed about, the relationship is the product. The paperwork is downstream of whether they trust you.
AI can help you prepare for that conversation. It can summarize what was said last time so you walk in informed. It can draft the follow-up so the note goes out today instead of Thursday. All of that is real value, and it works because a person still shows up.
What it cannot do is be the person. A seller in Lexington deciding what to do with her late mother's house is not looking for efficient correspondence. Automating that conversation does not scale your business. It hollows out the only part of it that was hard to copy.
Frequently asked
Questions people actually ask
Is it getting better at these things?
At some of them, meaningfully. Arithmetic handled by an attached calculator and web search for current facts are both real improvements. The core issue is unchanged: output is generated, fluency is not accuracy, and you cannot tell from the writing which case you are in.
Can I use it for a lease or a purchase agreement?
Use it to understand one. Paste a clause and ask what it means in plain English, what it obligates you to, and what to ask about. Do not use it to produce contract language you intend to sign. That comes from your attorney.
What about using it for tax questions?
Same answer. It is useful for understanding a concept before a conversation with your CPA, so you arrive with better questions. It is not a substitute for advice on your specific situation, and tax detail is exactly the kind of thing it renders confidently and imprecisely.
How do I catch errors I would not recognize?
By checking against the source rather than against your sense of whether it reads right. If it is a number, recompute it. If it is a rule, find the rule. If you cannot verify it and the cost of being wrong is real, do not use it.
Does this mean it is not worth using?
The opposite. A tool with known limits is a usable tool. What is not usable is a tool you believe has no limits, because you will find them at the worst possible moment.
Make your next move
A year from now, what will you be glad you started today?
You don't need another promise that everything will be easy. You need something useful to learn — and a next step you're willing to take.