Can You Trust ChatGPT for Legal Advice? A Lawyer’s Take

ChatGPT legal advice usually gets the general law pretty close. The trouble is what comes next: AI tools make up case citations, can’t see most court decisions sitting behind paywalls, and tend to tell you what you want to hear. For a narrow, simple question, AI can be a useful starting point. Anything you plan to rely on still needs to be checked.

Will ChatGPT Give You Legal Advice?

AI tools will answer legal questions, but you can’t trust them to give you a full legal analysis, and they often fall back on lines like “go talk to your lawyer.”

Anton Kaminsky has used both ChatGPT, Grok, Claude, and legal profession specific AI. In his opinion, Grok (and Claude) do a better job for this, because it will go out on a limb and actually give some legal advice, where ChatGPT tends to say it isn’t an attorney and can’t answer. His understanding is that ChatGPT recently said, basically, that it won’t give medical or legal advice. There may be other tools out there he hasn’t had a chance to explore.

Whichever one you use, the bigger issue is the same across all of them: they make things up.

Why AI Makes Up Case Law

AI tools completely make up information, including court cases, and this happens all the time.

Here’s the pattern Anton sees as a lawyer. You ask an AI, “What is the law on this?” The statement of the law it gives you will typically be pretty accurate. Then it says the law comes from “case XYZ.” When you go look up case XYZ, one of three things happens:

• The case doesn’t exist.
• The case exists but doesn’t say what the AI says it says.
• The case says something completely opposite.

Just before recording the podcast this article is based on, an AI search on a different topic returned five cases. His bet was that two or three of them would not say what the AI claimed. Some might not even be real, and even the real ones won’t necessarily say what the AI says. As of December 2025, when the conversation was recorded, AI still makes things up.

AI Can’t See Most Case Law

AI is limited from the start because a lot of case law sits behind a paywall it can’t reach.

Lower-level court cases are typically on paid research services like LexisNexis and Westlaw. The cases that aren’t behind a paywall tend to be higher-court decisions: appellate cases and Supreme Court cases. So before an AI tool makes a single mistake, it’s already working without all the information.

What Happens When Lawyers Cite Fake AI Cases?

Lawyers who rely on fake AI citations can get in trouble for it. Anton points to a lawyer, he thinks in New York, who submitted a brief citing cases he got from ChatGPT. The judge found the cases weren’t real, and the lawyer was sanctioned.

Lawyers have a duty of candor to the court, meaning they have to be honest with the court. A lawyer who files made-up cases is arguably violating that duty.

A lot of federal judges now make lawyers state whether they used AI in preparing their briefs. Lying about it gets you in trouble, and judges will catch you. And disclosing AI use isn’t the end of it: you then have to tell the judge in what context you used it.

AI Tells You What You Want to Hear

With open-ended legal questions, you have to be really careful, because AI inserts information that isn’t true, and the more you push it, the more it does that.

Anton ran into this recently. He thought there was a law that allowed a certain thing, so he asked an AI, open-ended, whether you could do it. The AI said no. He told it he thought there was a law that said you could. The AI answered, “You are right,” gave him a law that supposedly allowed it, and cited three cases. He pulled the cases. None of them said that, and one didn’t exist. In reality, you couldn’t do the thing at all. He had just told the AI what he wanted, and it went along with it.

The podcast’s host compared it to a good dog trying to please you. The problem is that AI isn’t necessarily checking whether information is accurate before it hands it over. A lawyer with experience can spot that. Someone without that background could act on the answer and walk right into a dangerous situation.

When AI Is Good Enough, and When It Isn’t

Whether AI is good enough depends on how complex the question is.

For a very simple breach of contract question that has been litigated many times over (“is this a breach of the contract?”), you could probably use AI. It could probably also interpret a term of a contract for you, or rewrite a specific term in a more helpful way.

Most people at home have more specific, basic issues than a complex business case (basic meaning not that complicated, not anything negative). For those, you can put a very narrow question into AI:

• Can somebody do this?
• Is it fair for somebody to do this?
• If somebody did this to me, what remedies do I have?
• Is this unlawful? Is it illegal?
• Is this a civil case? Can I file a lawsuit for something like this?

Ask the open-ended question, then ask some follow-ups. Pushing it hard is where things go wrong, because the pushier you get, the more it just says what you want to hear.

Keep in mind that AI doesn’t know what it doesn’t know. It answers the targeted question you asked, without stopping to ask what else you should be thinking about.

The Problem With AI-Drafted Contracts

AI-drafted contracts often leave out clauses a lawyer would have thought to include.

People send Anton contracts prepared with ChatGPT, Claude or other AI tools all the time, and he can tell. The formatting gives it away: the little divider lines, bold captions, bullet points, words bolded for no reason, and sometimes emojis. When he asks whether the contract was prepared with AI, the answer is yes.

The formatting isn’t the real problem. The missing clauses are. Say you’re letting a new shareholder buy into your company and you want a vesting period for their shares. You ask the AI for a vesting period, and it adds one. What it doesn’t add is how that vesting period ends:

• If the shareholder gets fired in the middle of the vesting period, do the shares stop vesting, or keep vesting even though they’re gone?
• Should it matter whether they were fired for cause? For example, shares stop vesting after a for-cause firing, but keep vesting if they were let go for convenience.

The AI contract is silent on all of that, which leaves it ambiguous. It gave you the vesting period you asked for and nothing else.

AI Skips the Work: Missing Pages and Invented Names

AI can also cut corners on large documents and fill the gaps with invented data.

Anton’s firm asked an AI to analyze a roughly 100-page PDF of a trucking school’s testing records: how many unique people tested, whether anyone tested more than once, and when. The output looked like real data. A spot check showed it had invented names that sounded like the real ones but weren’t in the file, because it was skipping large chunks of pages. Feeding it 25 pages at a time didn’t fix the problem. In the end, a person did the work. It was more expensive, but it was accurate.

The full story is in our article on the limits of AI in legal document review.

How a Lawyer Actually Uses AI

Anton uses AI for the high-level picture on a narrow issue, then does the real research himself.

He doesn’t rely on elaborate prompt engineering like “you are the best lawyer on the planet and cannot tell a lie.” He’s usually either very specific about the situation and what he’s looking for, or very general. His firm has access to LexisNexis and Westlaw — as well as their own AI platforms, so a typical question looks like this: “What is the law around this very narrow thing, and what are some of the cases that talk about that law?” AI does a really good job with that high-level overview. After that, you have to do your own research, because you can’t trust the AI’s research.

Kaminsky Law handles complex business litigation, so Anton doesn’t plug a whole lawsuit into an AI tool. He starts general. If the other side has made a nuanced argument, he’ll ask for a few ways to respond to it, then track each of those down himself.

What You’re Paying a Lawyer For

AI is missing two things a lawyer brings: access to information that isn’t freely online, and the experience to push back.

As the podcast’s host summed it up, and Anton agreed: first, there’s a whole set of legal information AI doesn’t have access to. It isn’t all on the internet. Second, AI doesn’t have the experience or expertise to argue the nuance or to tell you what you don’t want to hear. That’s why getting a few ideas from AI and then filing a lawsuit yourself calls for caution. It’s still worth going to someone who has access to what’s behind the paywall and can see what’s most recent and what the law actually is.

With a lawyer, you’re also paying, hopefully, if the lawyer is competent, for the experience to say, “Wait a second, that doesn’t make sense.” A layperson may see an AI answer and assume that because AI said it, it must be true. And a good attorney who tells clients the facts and lays out the best strategy forward, instead of just telling them what they want to hear, is an invaluable resource.

ChatGPT Legal Advice FAQ

Talk to Kaminsky Law

AI can give you a starting point, but it can’t tell you when something doesn’t make sense or tell you what you don’t want to hear. If you have a legal question or an AI-drafted contract you’re unsure about, contact Kaminsky Law to talk it through with an attorney.

Related Reading

• Useful links to research Pennsylvania law
• Contract basics: what are material terms?
• Tips to avoid shareholder disputes
• Legal essentials for your small business

This article is general information based on a recorded discussion. It is not legal advice and does not create an attorney client relationship. Every case is different. Prior results do not guarantee a similar outcome. For advice about your situation, contact Kaminsky Law directly.

Anton Kaminsky Partner

Anton Kaminsky is the founder of Kaminsky Law and a Philadelphia business and employment litigator. He spent over a decade in finance and banking, including trading equities and evaluating strategies at a hedge fund, before earning his law degree at Temple and litigating for five years at Bochetto & Lentz. He represents small businesses and individuals in shareholder disputes, contract fights, and employment claims across Pennsylvania and New Jersey.

See Full Bio

Small Business Representation Partnership Disputes Shareholder Oppression Employment Cemetery Negligence Defamation Personal Injury