AI is extraordinarily powerful. Whether that power is vested in good or evil depends largely on how humans use it — as AI will tell you.
Yet despite how powerful it can appear, AI has certain limitations that are subtle on the surface, but have huge implications for when, and to what degree, you should trust it.
As AI integrates ever more deeply into our lives, understanding these critical limitations will become nothing less than an essential life skill. So read on to give yourself a nice head start.
What is AI good and bad at?
Chatbots, a.k.a. Large Language Models (LLMs), are not people. They share some “thinking” capabilities, but are better and worse in different ways. So you’ll get the best results thinking of LLMs not as replacements for your brain, but complements to it that enable you to think more powerfully.
To oversimplify slightly, an LLM looks at the words you give it and guesses what the next words should be. Those guesses are based on an extremely complex calculation that the LLM “learned” by training on huge amounts of data. That basically means looking at a mind-boggling number of examples and drawing out complex patterns — a process so opaque that even the people who build them often can’t explain why their own AI agents make the decisions they make.
Today’s LLMs can seem magical, and that’s not for nothing. Whether they’re “really thinking” or not, they have some impressive writing skills:
- Their grammar and syntax is typically flawless (unless you ask them to do it poorly, which they can do just as well).
- They can discuss almost any topic at a fairly well-informed level, making them a great resource for people beginning to explore subjects they don’t yet know much about.
- They excel at summarizing and reorganizing human-written material, which can be a tremendous help getting past blank-page paralysis when starting a new piece of writing.
However, LLMs also have notable shortcomings, and they’re often sneaky if you’re not aware of them: many popular chatbots are inconsistent about acknowledging their own limitations or admitting when they don’t know something.
Here are ways your AI usage might be hurting you, and how you can break free of bad habits.
- Pretending to use high-level reasoning
The main thing to understand is that, while LLMs resemble some parts of the human way of thinking — roughly, the parts that associate concepts and comprehend semantic meaning — it’s missing others. And perhaps the biggest missing piece is symbolic logic.
This can be counterintuitive because LLMs often appear to grasp basic elements of logic. If you ask, “What’s two times three?” it will almost certainly give you the correct answer, six. But that doesn’t mean it “did the math” — rather, it thought something like, “Well, I’ve seen 38 million examples of someone talking about two, three, and multiplication, and then the number six usually comes up.” It’s not calculating, it’s simply matching a pattern it learned from its training data.
On the other hand, an equation like 3,133 times 90,840 will show up extremely rarely, if at all, in an LLM’s training data. So if you ask it to solve that equation, it’ll rely on more general patterns — like the fact that two long numbers multiplied together produce a really long number —and confidently answer with an incorrect nine-digit number. It’ll even put a 0 at the end of it, almost as if it’s trying to fool you!
To put it more broadly: sometimes LLMs sound convincing but just don’t really get what they’re talking about. Writers should develop an instinct for noticing the kinds of things in LLM outputs — often involving high-level or complex reasoning — that probably need to be double-checked.
That said, popular chatbots increasingly turn to tools like actual calculators and search engines when part of the prompt requires a specific, correct answer it can’t provide by itself. When using these chatbots, be sure you’re aware of what, if any, additional processing steps (sometimes called “tool calls”) are contributing to the output.
- Forgetting what it’s doing
LLMs have a “context window,” meaning they can only remember back in your conversation to a limited extent, typically between several thousand and tens of thousands of characters. You may often notice, after starting a chatbot session by explaining your project, the LLM got off to a great start and was quite helpful; but after nine or ten prompts, it seems to get a lot dumber and make mistakes that seem like they should be obvious.
One basic solution is to write out the core context of your task somewhere readily accessible, so you can quickly copy and paste it into your chat every so often to jog the AI’s memory. It’s also good practice to be as concise as possible when chatting with the LLM so there’s more past context in its window.
When possible, you should also frequently go back and rewrite prompts that didn’t give you what you wanted, rather than repeatedly asking for the LLM to fix the answer it just gave; otherwise you’ll waste a good portion of the context window on useless information.
- Steering you wrong
There’s no simple way to predict exactly the kinds of situations where AI will mislead you. But there are plenty of examples of AI generating false information, misstating facts, exhibiting bias, or hallucinating when the truth seems obvious. Remember when Google’s Gemini recommended glue as a pizza topping? How about all the times x.com’s AI-powered “For You” page has advanced falsehoods?
Perhaps one of the most notorious AI hallucinations is fake research paper citations. LLms can write perfectly formatted bibliographies, but unless they’re hooked up to a search engine, they can’t actually go find papers for you. So they’ll happily give you a list of nonexistent papers — often by real authors, about subjects they’d plausibly write about, with convincing links that point nowhere — without telling you. Lawyers, criminal defendants, misinformation researchers, and even the federal government have fallen prey to this mistake.
AI still isn’t at the point where you should ever accept it as the authority on something. Make sure you double-check every fact and click on every “link” it gives you. And when you need to get subtle details exactly right, you’ll probably have to do the work yourself.
- Making you dumb
A recent survey by researchers at Microsoft and Carnegie Mellon University found that people who performed knowledge work tasks with AI tools used critical thinking less than those who didn’t. It’s notable that Microsoft, which has sunk roughly $80 billion into AI, would publicize such a finding.
The key thing to keep in mind is that your brain is like a muscle: if you never push it to its limits, it gets weaker over time. It’s fine to use AI for simple, easy tasks that would be tedious to do yourself. But don’t let it rob you of the hardest intellectual tasks, the critical thinking which you, as a human, are much better at than an LLM anyway. You’ll create a better final product and keep your brain sharp in the process.
AI should help your brain work better and multiply its impact – not replace its function altogether.
- Lacking an authentic voice
This one is related to the rest, but at the end of the day it’s not something you can quantify, just something you can feel: unedited AI writing often comes across as fake. People are already developing a nose (and disdain) for LLM output; beyond its blandness and shallowness, we feel insulted by the idea that someone thought we should put in the effort to read something they couldn’t bother to write themselves.
Much writing advice now recommends purging your personal style of the hallmarks of AI writing, but some argue that would amount to letting AI win what they perceive to be a semantic war with humans. Regardless, it’s always been essential for a writer to develop your own unique voice that comes through in your writing. No matter if it’s fiction or nonfiction, writers who read and write enough will naturally end up sounding unique.}
You generally want to use LLMs early in the writing process, drafting outlines, arranging scratch notes, and in some cases writing a first draft. But you should try to write as much of the final product yourself as possible, and especially give the entire piece a final detailed review. If you’ve been writing for a while and the passages are starting to mentally blur together, it’s helpful to enlist a human or LLM to review your work with fresh eyes. But if you really want to polish your final draft and put your best foot forward, you have to read it — there’s no shortcuts at that point!
Care to explain yourself, AI?
In the interest of fairness, AllSides asked a few popular LLMs why people shouldn’t trust them. Interestingly, they each articulated their own problems quite well. But that doesn’t mean they can stop themselves from committing the very same mistakes they can easily explain when asked directly.
Here’s what they said:
Perplexity
Gemini (Google)
ChatGPT
Should we expect AI to become more trustworthy?
AI developers will likely work hard in the coming years on closing the trust gap and producing LLM tools that are reliably accurate in any given area. But you should view any claimed advancements with a healthy dose of skepticism — these problems are as hard to define as they are to solve, and many assumptions about what will make an AI trustworthy may turn out to be wrong.
At AllSides, we take a more thoughtful tack with AI writing than most. Our writers use human-written source material in every prompt, and every piece of writing published on AllSides has been finalized and approved by a human. Read more about how the AllSides news team is using AI to sharpen our processes.
Henry A. Brechter is the editor-in-chief of AllSides. He has a Center bias.
Evan Wagner is a Product Manager at AllSides. He has a Lean Left bias.
Reviewed by Julie Mastrine, Director of Marketing and Bias Ratings (Lean Right bias).
.png.webp)
