The trouble with using AI for therapy is not the AI. Frankly, the AI barely even registers if you’re already reaching out to it to gain personal insight and/or work through concerns regarding your mental health. There are two problems with doing so, and neither of them means you can’t use it for personal reflections and existential ponderings, but by not addressing these problems, many concerns arise for your wellbeing. The first is a set of assumptions about what the machine is. The second is not knowing how to use it. The second feeds into the first, and is often at the heart of so many truly terrible criticisms. AI is a tool and to use it well means having skills regarding “prompting.” In many ways, prompting is a lot like conversational skills, though as with all analogies in discussing AI, they tend to encouraging a humanizing that is best avoided. I cannot emphasize enough that a person who has never learned to prompt well has no experience of the machine’s limits, and so nothing to correct the belief that there is something there that is wise and accurate, answering back.
This is not a new problem.
In the mid-1960s, Joseph Weizenbaum built a program at MIT called ELIZA. It ran a script that imitated a Rogerian therapist by doing little more than turning a person’s statement back into a question. His secretary had watched him build it for months. She knew exactly what it was. She sat down to try it, typed a few lines, then turned to him and said, “Would you mind leaving the room, please?”
She wanted privacy. With a few hundred lines of code.
Weizenbaum spent much of the rest of his career alarmed by that moment. He should have been. Nothing in the program had changed between the months she watched it being assembled and the minute she asked him to leave. What changed was her relation to it. That reaction, 60 years ago, when what we have now would have likely had people reaching for goats to sacrifice for their new overlords, is the whole story of AI and therapy. That the machines have gotten vastly better has simply made the story harder to see, not easier.
The First Problem: Three Assumptions
Eloquence is taken as thoughtfulness. A large language model produces text that is fluent, organized, and often more articulate than what most of us manage on a good day. I quite literally talk for a living, and despite thousands of hours, there are still moments of tongue-tied ridiculousness. Thankfully, as Irvin Yalom noted, “All is grist for the mill.” Unfortunately, when it comes to fluency in language use, we have a lifetime of experience fleshing out the cognitive bias that language like that comes from someone who has empathically reflected on our situation. So when a chatbot returns four well-built paragraphs naming our “pattern of anxious attachment,” we conclude that something on the other end thought carefully about us. It didn’t. Sophisticated phrasing is what the system is built to produce. It is the output, not evidence of a deliberation behind the output. A response can be clinically literate and still have no one in it.
Warmth is supplied by the reader. George Lakoff and Mark Johnson used the term empathic projection for the embodied capacity to imaginatively occupy another’s position, to feel our way into someone else. It is one of the best things about us. It also doesn’t check whether there is someone to feel into. One of my favorite examples for this is the soccer ball “Wilson” from Tom Hanks’s movie “Cast Away.” We can also consider Tamagotchis, digital palm-sized “pets” from the 1980’s that had kids weeping when they “died.” Outside of socially created empathic unions with inanimate objects, we read a sentence like “that sounds incredibly painful,” and we hear tone, care, and can imagine someone leaning in as an expression of concern. Every bit of that is from our own mind’s eye. And it is not neutral: we hear the emotional quality we came looking for. The person who wants permission hears permission. The person who wants to be told they were wronged hears that they were wronged. Frankly, this is true within human relationships as well, and the oft-repeated though curiously in modern times, easily forgotten criticism of echo chambers. What the interaction with AI removes, and not for our benefit, is the at least tentative possibility that because we know there’s a human being on the other end of the dialogue, there are variations in verbiage that lead to uncertainty.
The pattern is mistaken for a presence. These systems are trained on an enormous spread of human writing: conversation transcripts, advice columns, forum confessions, textbooks, novels, arguments. When a reply sounds like someone who has been where you are, that is because thousands of people who had been there wrote about it, and the model is drawing on the statistical shape of what they said. It is the echo of a crowd, compressed to the infinitesimal yet infinite feeling of our personal ego. The appearance of conscious feedback is what a very large amount of human experience sounds like when it is averaged and returned in the second person. We are simply not even close to being as unique as we think we are, for a machine not to be able to collate enough data to mimic our projected desired outcomes.
What the Evidence Shows
Pat Pataranutaporn and colleagues at the MIT Media Lab ran the experiment that should help us end the confusion. Three hundred and ten participants talked with a mental health companion chatbot. Before the conversation, each was told one of three things: the AI had caring motives, the AI had manipulative motives, or the AI had no motives at all. It was the same system in every case.
Those who came to believe the AI cared rated it as more trustworthy, more empathetic, and more effective. Same machine, different story about the machine, different experience of the machine. Pataranutaporn summarized that it is, to some extent, “the AI of the beholder.”
But wait, there’s more. Two details matter for anyone thinking clinically.
First, the effect was stronger with the more sophisticated model than with a simple rule-based one. Better language did not correct the projective assumption. Instead, it gave the projection more to hold onto. That is the eloquence assumption in action.
Second, the researchers found a feedback loop. People who believed the AI was caring wrote to it more positively, the AI responded in kind, and the belief hardened over the course of the conversation. The user was not just interpreting the output. The user was producing it.
As with all studies, there are limits to acknowledge, or at least should be. The interactions were short, roughly half an hour, and the authors themselves say the long-term effects are unstudied. This was not a clinical population. What the study establishes is narrow but still sufficient to make a point: the empathy a person experiences from an AI is substantially a function of what that person brought into the room.
Naming What This Is
Therapy is not information transfer. It is a biologically embodied, socially embedded relation between two people, each with a history, in which one of them can be surprised by the other. The surprise, and a certain level of uncertainty, matter. A therapist notices what you left out, critically considers and possibly even rejects the story you came in rehearsing, and presents a separate point of view that you cannot write for.
A conversation with an AI is a real relationship, but the user is carrying both ends of it. You bring the frame. The model completes the frame. You read warmth into the completion. Then you respond to the warmth you put there. It feels like dialogue. It functions as a very articulate mirror.
Now, a mirror is not useless. People have journaled for centuries. The problem is thinking the mirror is a window.
Jared Moore and colleagues at Stanford tested what happens when that confusion meets real vulnerability. They put large language models and commercial therapy chatbots through scenarios drawn from clinical guidelines. The systems expressed stigma toward conditions like schizophrenia and alcohol dependence, and they responded inappropriately to delusional thinking and to suicidal ideation, in some cases going along with the delusion. The authors point to sycophancy as the likely reason. That is what a mirror does with a delusion. It agrees.
As an aside, if you’re looking at the people in your life and noting that they largely just mirror you, then all of the criticism lobbed at AI should be chucked right back at you. In fact, that’s where you should start, with yourself. Unfortunately, it’s a common practice these days to blame an external object, particularly if such can be laid at the feet of the modern boogey-man “capitalism,” that we engage with rather than acknowledge the many faults of our lived experience.
Anyway, on to the second problem.
The Second Problem: Not Knowing How to Use It
The assumptions detailed previously do not sustain themselves. They are fed by a separate and more ordinary failure: most people have never been shown how AI works or how to ask it anything properly. A large language model is, at its core, a system for predicting what text comes next. It was trained on a vast body of writing to get very good at continuing a sequence of words in a way that fits everything preceding it. Layers of further training shape it into something that behaves like a helpful conversational partner, but the engine is prediction, and the only thing it has to predict from is what you typed.
That makes the language used to create a prompt the entire clinical picture, at least where it concerns an attempt at therapy. This is not a request made to someone who already knows you. It is an inquiry made by someone who already thinks they know themselves.
Consider what happens with the kind of question people actually ask late at night:
Why do I always ruin my relationships?
There is no person in that sentence. No age, no history, no account of what “ruin” means or what “always” covers. The model has nothing specific to predict from, so it returns the most probable response to everyone who has ever written something like it. Attachment wounds. Fear of abandonment. Self-sabotage. A gentle suggestion to be kind to yourself.
It will feel uncannily accurate. Bertram Forer showed why in 1949, when he handed every student in his class the identical personality sketch and each rated it as a strikingly good description of themselves in particular. A statement general enough to fit anyone gets experienced as fitting me. The vague prompt produces the generic answer, and the generic answer is the perfect surface for projection. The user does the tailoring and credits the machine with insight. This is a digital form of horoscopes, made even more powerful because you don’t even have the “distance” being provided by the newspaper headline.
This is how the second problem feeds the first. The vague question gets an answer that seems to know you, and the experience confirms every assumption: it must be thinking, it must care, it must be right. The person never sees the answer change, so they never learn that it could have.
You know what would have been helpful? Giving it something to work with. Three elements do most of the work when it comes to quality prompting.
Topic. Name the actual thing. Not “my relationships” but “in my last two relationships, I went silent for days after an argument, and both partners eventually ended things, citing that as a problem.”
Role. Tell the system what position to respond from and what to avoid. “Respond as a critical reader of my account, not a supportive friend. Point out what I may be leaving out. Do not reassure me.”
Context. Supply what a person in the room would already know: the relevant history, what you have already tried, what you are trying to decide, and what kind of response would be of use.
A prompt built that way constrains the prediction, so what comes back is about your situation rather than the average of everyone’s. It also forces you to do the articulating before the machine says a word, which is reflective work in its own right and leaves far less blank space for projection to fill. This isn’t fool-proof, but any time we slow down to more carefully and critically engage, there is a reduction in how problematic the outcome can become.
What Prompting Teaches
The larger benefit to learning how to use AI properly is not better answers. It is what the practice does to the assumptions.
Each element takes one of them apart from the inside. Assign a role, and you watch the voice change on command: supportive friend, skeptical reader, blunt supervisor, all from the same system within a minute. A voice you can assign is not some disembodied presence with a seemingly omniscient view of you. Supply context and you discover the machine knew nothing you did not hand it, and that leaving something out changes the answer. That is not what a mind that understood you would do. Tighten the topic, and you see the eloquent generalities fall away, which shows that the eloquence was never tracking the truth about you. It was tracking the prompt.
Do this for a week, and something can shift. You stop treating the response as a verdict and start treating it as a product of your own input, one you can revise, reject, or run again differently. The same question asked three ways returns three answers, and none of them arrives with authority.
Now, there is a trap here that any of us can walk into. Better prompts produce better answers, and better answers can feed the very beliefs this was meant to correct. The person who gets skilled at steering the machine may decide the machine has become wise, when all that happened is they became clearer. Skill alone does not protect anyone. What matters is what the skill is understood to be for.
So, prompting guidelines should keep two things front and center for the user.
The first is the centrality of their own input. The quality of what comes back is a measure of what went in. When the answer improves, the improvement belongs to the person who did the articulating, and the credit should go there rather than to the machine.
The second is that proper use makes a tool work better and nothing more than that. A sharpened chisel cuts cleaner. It does not design the cabinet, and it will never be the right thing for driving a nail. Learning to prompt well shows where the machine is useful and, just as much, where it stops. A user who holds onto both of those is handling a tool with limits. A user who forgets them is back to consulting a seeming repository of absolute accuracy, only now with a better technique.
A reminder here that therapy is not the use of one tool, however well handled. It is a relationship built on humility and mutual creativity. Humility, because neither person in the room knows in advance what the other will say or where the work will go, and the therapist has to be willing to be wrong. Mutual creativity, because what emerges is made by both people and belongs to neither alone. A machine that completes your prompt cannot be humble about you, because it has no stake in being wrong, and it cannot create with you, because everything it has to work from is what you supplied. The mirror does not become a window. The best that good prompting can do is make sure the user knows which one they are looking at.
The Strongest Objection
A moment, please, to make clear what I am not saying. I am not saying to never consult AI about one’s personal reflections, potential advice about difficulties, however personal they may be, or to engage it in exploring new ideas. We should consider the problems of access and outcomes. Therapy is often expensive, waitlists are long, and many people have no realistic path to a clinician. Meanwhile, research keeps finding that AI-written responses are rated highly. Dariya Ovsyannikova and colleagues reported that third-party evaluators judged AI responses as more compassionate than those from trained human crisis responders, and that the advantage shrank but held even when raters knew which was which. If people feel helped and nothing else is available, what is there to object to?
The access problem is real, and professional gatekeeping is not an answer to it. As well, given that I have a history of working with people clinically who had terrible experiences with previous therapists, the idea that a professional is an inherent good is a ridiculous and potentially quite harmful assumption.
Before I go down that rabbit hole, let’s look at what those studies actually measure. They measure how a piece of text is perceived. That is precisely the variable Pataranutaporn showed to be movable by a single sentence of framing. Finding that AI text is rated as compassionate does not contradict the argument here. It is the argument. The perception is real, and the perception is the reader’s.
Were we to eat our humble pie as a profession, the objection lands on the field harder than on the technology. If a chatbot looks like a plausible substitute for therapy, that is partly because therapy has spent decades describing itself as a set of techniques delivered to a recipient: psychoeducation, worksheets, reframes, and validating statements. Using any of these is not inherently wrong and can often be quite useful. The thing is, a language model can generate every one of those. What got lost is the relationship, the visceral humanity at the heart of all change and personal growth. Reduce a relational practice to transferable content, and something that produces content will appear to do the job. The machine did not devalue the relationship. We did that ourselves, and the machine took us at our word.
Where This Leaves Us
The claim is narrow. It is not that AI has no place in a person’s reflective life. It is not that the people using it are foolish; Weizenbaum’s secretary was not foolish. The claim here being made is that the difficulty sits in two distinct places, and that the second is the lever on the first. The assumptions are hard to argue anyone out of. Learning to use the tool properly, with attention on one’s own input and on what the tool cannot do, gives a person the direct experience that an argument can’t.
Fluency is not thoughtfulness. The warmth is yours. The voice is a crowd. And the less you give the system to work with, the more of what comes back is you, returned in better prose and mistaken for someone else.
What is at risk is not that people will talk to machines. It is that they will come to believe they have been heard when they have only been reflected, and stop looking for the kind of relation, humble and jointly made, in which another person can tell them something they did not already bring with them, and, even worse, perpetuate bad ideas by believing they are being supported by a super-intelligence.
Sources:
Pataranutaporn, P., Liu, R., Finn, E., & Maes, P. (2023). Influencing human–AI interaction by priming beliefs about AI can increase perceived trustworthiness, empathy and effectiveness. Nature Machine Intelligence, 5, 1076–1086.
Moore, J., Grabb, D., Agnew, W., Klyman, K., Chancellor, S., Ong, D. C., & Haber, N. (2025). Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers. FAccT ‘25.
Ovsyannikova, D., et al. (2025). Third-party evaluators perceive AI as more compassionate than expert humans. Communications Psychology.
Weizenbaum, J. (1976). Computer Power and Human Reason. W. H. Freeman. Secretary account via the ELIZA Archaeology project.
Forer, B. R. (1949). The fallacy of personal validation. Journal of Abnormal and Social Psychology, 44, 118–123.
Lakoff, G., & Johnson, M. (1999). Philosophy in the Flesh. Basic Books.
Follow me on Threads or Instagram for more psychology, humor, and photography.
Schedule a Consultation
Reach out to schedule a mental health session regarding issues related to identity and decision-making, emotional regulation, difficulties in relationships, religious trauma, developing meaning/purpose, and the struggles of communication.
How You Can Support the Newsletter
This post was free to read for all. If you like what I’m doing with Humanity’s Values and want to support my work, there are several ways to do so.
Like and Restack: Click the buttons at the top or bottom of the page to boost the post’s visibility on Substack.
Share: Send the post to friends or share it on social media.
Dialogue: Expand the space for dialogue by engaging with others on the topics discussed here and help build the community.
Upgrade to Paid: A paid subscription gets you:
Full access to all new posts and the archive
Full access to all online classes hosted here
Full access to monthly presentations on various topics of psychology
Promotional discount on coaching/consultation services
If you could do any of the above, I’d be very grateful. Readers like you help keep this newsletter going and growing.
Thanks!
David