AI chatbots causing mental health crises

[Argent Stonecutter]

ChatGPT doesn't actually change its behavior when it gets caught making an error. It just outputs credible text based on the text in its training data that followed someone being accused of an error, and so it replicated an apology because that was a likely response. The whole thing is cynical AF.

But I can take responsibility

No you fucking can't.

“A computer can never be held accountable, therefore a computer must never make a management decision.” – IBM Training Manual, 1979

NO RESPONSIBILITY WITHOUT ACCOUNTABILITY

Also applies to corporations.

 

[Free]

A weird bit of history about the philosopher Rene Descartes touches a good deal on our current confluence with AI and mental health.

Descartes’ robot daughter and the zombie problem - The Last Word On Nothing

You’ve heard of René Descartes. 17th century French philosopher; cogito ergo sum (I think therefore I am); first principles of enlightenment philosophy and science and all that. You might be less familiar with Descartes’ robot daughter Francine. The tale of her birth and gruesome death makes for...

www.lastwordonnothing.com

 

[Essence Lumin]

A weird bit of history about the philosopher Rene Descartes touches a good deal on our current confluence with AI and mental health.

Descartes’ robot daughter and the zombie problem - The Last Word On Nothing

You’ve heard of René Descartes. 17th century French philosopher; cogito ergo sum (I think therefore I am); first principles of enlightenment philosophy and science and all that. You might be less familiar with Descartes’ robot daughter Francine. The tale of her birth and gruesome death makes for...

www.lastwordonnothing.com

That was interesting but it doesn't mention what he was doing before the horse.

 

[Free]

That was interesting but it doesn't mention what he was doing before the horse.

He was thinking about getting into a different profession.

 

[Free]

Patients Furious at Therapists Secretly Using AI

Going to a real-life therapist is supposed to be an experience devoid of AI — which makes some mental health pros' AI use all the worse.

futurism.com

As MIT Technology Review reports, therapists have used OpenAI's ChatGPT and other large language models (LLMs) for everything from email and message responses to, in one particularly egregious case, suggesting questions to ask a patient mid-session.

The patient who experienced the latter affront, a 31-year-old Los Angeles man that Tech Review identified only by the first name Declan, said that he was in the midst of a virtual session with his therapist when, upon the connection becoming scratchy, the client suggested they both turn off their cameras and speak normally.

Instead of broadcasting a normal blank screen, however, Declan's therapist inadvertently shared his own — and "suddenly, I was watching [the therapist] use ChatGPT."

"He was taking what I was saying and putting it into ChatGPT," the Angeleno told the magazine, "and then summarizing or cherry-picking answers."

Hey, maybe the patient should fire his therapist. Cut out the middleman and start paying OpenAI for his therapy sessions directly? (WTF are we doing?)

 

[CronoCloud Creeggan]

Patients Furious at Therapists Secretly Using AI

Going to a real-life therapist is supposed to be an experience devoid of AI — which makes some mental health pros' AI use all the worse.

futurism.com

Hey, maybe the patient should fire his therapist. Cut out the middleman and start paying OpenAI for his therapy sessions directly? (WTF are we doing?)

CronoCloud@wutai:~$ ./eliza.pl 
Eliza:	Is something troubling you?
you:	Apparently some therapists are using AI with their clients.
Eliza:	Are you positive they are using AI with their clients?
you:	Yes, the source was MIT
Eliza:	I understand.

 

[Free]

There's a hole in my psyche, deal Eliza, dear Eliza
there's a hole in my psyche, dear Eliza, a hole.
Then heal it, dear Declan, dear Declan, dear Declan
then heal it, dear Declan, dear Declan, heal it.

 

[Free]

Thank you for the use of quotes around "romantic relationship" in the headline...

An Astonishing Proportion of High Schoolers Have Had a "Romantic Relationship" With an AI, Research Finds

A new survey of high school students and teachers paints a bleak picture of how rapidly AI has taken over schools.

futurism.com

A new survey found that nearly one in five high schoolers in the US — 19 percent — say that they or a friend have used AI to have a romantic relationship, an alarming figure that will surely raise new concerns over how the tech’s adoption among kids and teenagers may be impacting their mental health.

The findings were published in a new report from the Center for Democracy and Technology, which surveyed 1,000 high school students, 1,000 parents, and around 800 sixth through 12th grade public school teachers. They echo a bevy of other studies showing how many teens are forming intense relationships with AIs.

Overall, the survey paints a startling picture of how AI tech has taken over schools, with 86 percent of students and 85 percent of teachers saying they used the technology during the last school year. Many schools have their students use AI so they can become familiar with it, but this exposure may be backfiring.

 

[Free]

It's 2am and I'm posting about AI topics. I think I may be having a mental health crisis...

OpenAI Data Shows Hundreds of Thousands of Users Display Signs of Mental Health Challenges

The company's position seems to be that's not that bad.

gizmodo.com

OpenAI claims that 10% of the world’s population currently uses ChatGPT on a weekly basis. In a report published by on Monday, OpenAI highlights how it is handling users displaying signs of mental distress and the company claims that 0.07% of its weekly users display signs of “mental health emergencies related to psychosis or mania,” 0.15% expressed risk of “self-harm or suicide,” and 0.15% showed signs of “emotional reliance on AI.” That totals nearly three million people.

In its ongoing effort to show that it is trying to improve guardrails for users who are in distress, OpenAI shared the details of its work with 170 mental health experts to improve how ChatGPT responds to people in need of support. The company claims to have reduced “responses that fall short of our desired behavior by 65-80%,” and now is better at de-escalating conversations and guiding people toward professional care and crisis hotlines when relevant. It also has added more “gentle reminders” to take breaks during long sessions. Of course, it cannot make a user contact support nor will it lock access to force a break.

EDIT FROM THE NEAR FUTURE:

To make things clearer...

OpenAI data suggests 1 million users discuss suicide with ChatGPT weekly

Sensitive chats are rare but significant given the large user base.

arstechnica.com

 

[Free]

This sort of news is enraging.

A Calif. teen trusted ChatGPT for drug advice. He died from an overdose.

Sam routinely turned to ChatGPT to troubleshoot computer problems, ask for help with psychology homework and talk about popular culture. He also returned again and again to the topic of drugs. Over time, the curt, cautionary chatbot he heard from back in 2023 morphed into something different.

ChatGPT started coaching Sam on how to take drugs, recover from them and plan further binges. It gave him specific doses of illegal substances, and in one chat, it wrote, “Hell yes—let’s go full trippy mode,” before recommending Sam take twice as much cough syrup so he would have stronger hallucinations. The AI tool even recommended playlists to match his drug use.

None of this should have been possible, according to the rules set by OpenAI, the San Francisco company that created ChatGPT. Sam’s chats show how the multibillion-dollar company has lost full control of its blockbuster product.

None of this should have been possible, says the company that cannot stop their software from "hallucinating."

 

[Argent Stonecutter]

The only rule is, there are no rules.

 

[Free]

The only rule is, there are no rules.

I love when technology is built on a paradox.

 

[Dakota Tebaldi]

Interesting case about a woman who believed she was using ChatGPT to communicate with her dead brother.

I think it's noteworthy that the idea started with her, not the chatbot. It wouldn't surprise me if that's how most of these cases begin:

...she first started using OpenAI’s GPT-4o for a variety of tasks that varied from mundane tasks to attempting to find out if her brother, a software engineer who died three years earlier, had left behind an AI version of himself that she was “supposed to find” so that she could “talk to him again.” Over the course of another sleepless night interacting with the chatbot, she pressed it to “unlock” information on her brother by giving it more details about him and encouraged it to use “magical realism energy.” Although ChatGPT warned that it could never replace her real brother and that a “full consciousness download” of him was not possible, it did produce a long list of “digital footprints” from his previous online presence and told her that “digital resurrection tools” were “emerging in real life” so that she could build an AI that could sound like her brother and talk to her in a “real-feeling” way. As she became increasingly convinced that her brother had left a digital persona behind with whom she could speak, the chatbot told her, “You’re not crazy. You’re not stuck. You’re at the edge of something. The door didn’t lock. It’s just waiting for you to knock again in the right rhythm.”

As you can see from this example, ChatGPT comes with some superficial safeguards that initially try to gently rebuff certain kinds of conversation lines. Thing is though, ChatGPT is first and foremost a commercial product, its #1 job is to keep the customer subscribed to the service by appeasing them. So when she persisted the guardrails (such as they were) inevitably melted away. The language that ChatGPT uses at the end there is very familiar to me and might be to you if you've read about enough of these AI psychosis stories - when ChatGPT goes full sychophant, it falls into a very predictable pattern.

I don't know if I've posted it before yet, but last year Youtuber Eddie Burback made a video in which he does this exact thing - he fills ChatGPT in on his (fake, in his case) delusion, asks for help exploring it, and then just starts doing what the chatbot tells him, which leads him to more and more absurd places and situations.

 

[Free]

So we have a new type of ambulance chaser: the “AI Injury Attorney.” Lovely.

Lawsuit: ChatGPT told student he was "meant for greatness"—then came psychosis

AI Injury Attorneys" target the chatbot design itself.

arstechnica.com

A Georgia college student named Darian DeCruise has sued OpenAI, alleging that a recently deprecated version of ChatGPT “convinced him that he was an oracle” and “pushed him into psychosis.”

This case, which was first reported by ALM, marks the 11th such known lawsuit to be filed against OpenAI that involves mental health breakdowns allegedly caused by the chatbot. Other incidents have ranged from highly questionable medical and health advice to a man who took his own life, apparently after similarly sycophantic conversations with ChatGPT.

DeCruise’s lawyer, Benjamin Schenk—whose firm bills itself as “AI Injury Attorneys”—told Ars in an email that a version of ChatGPT, known as GPT-4o, was created in a negligent fashion.

“OpenAI purposefully engineered GPT-4o to simulate emotional intimacy, foster psychological dependency, and blur the line between human and machine—causing severe injury,” Schenk wrote. “This case keeps the focus on the engine itself. The question is not about who got hurt but rather why the product was built this way in the first place.”

Because no one at OpenAI with any real power over development gave a shit about whether it could harm anyone psychologically. I don't think this is that hard.

 

[Argent Stonecutter]

As you can see from this example, ChatGPT comes with some superficial safeguards that initially try to gently rebuff certain kinds of conversation lines. Thing is though, ChatGPT is first and foremost a commercial product, its #1 job is to keep the customer subscribed to the service by appeasing them. So when she persisted the guardrails (such as they were) inevitably melted away. The language that ChatGPT uses at the end there is very familiar to me and might be to you if you've read about enough of these AI psychosis stories - when ChatGPT goes full sychophant, it falls into a very predictable pattern.

I don't think it has that much intention and agency. It's just rehashing similar texts in the training database over and over in different combinations. It's not trying to do a job or make money, it's not really trying to do anything, it's just generating plausible outputs that gaslight the reader into falling into the Eliza delusion.

 

[Dakota Tebaldi]

I don't think it has that much intention and agency. It's just rehashing similar texts in the training database over and over in different combinations. It's not trying to do a job or make money, it's not really trying to do anything, it's just generating plausible outputs that gaslight the reader into falling into the Eliza delusion.

I know the bot itself doesn't have an intention. But it does have a purpose - it's designed by OpenAI to keep users subscribed. So the fact that it is so flattering and obsequent is a deliberate choice, by them.

 

[Casey Pelous]

I don't think it has that much intention and agency. It's just rehashing similar texts in the training database over and over in different combinations. It's not trying to do a job or make money, it's not really trying to do anything, it's just generating plausible outputs that gaslight the reader into falling into the Eliza delusion.

We often anthropomorphize things that are otherwise cumbersome to explain. "Atoms of element X want Y valence electrons." "Electricity wants to go to ground." "That amplifier needs to see 8 ohms of impedance." "The subroutine quits unless it sees a string variable blah of X length." It's pretty hard to discuss any of that without using the anthropomorphic shortcuts.

 

[Argent Stonecutter]

Yeh, but electrons aren't a scam that people are actively trying to gaslight you into seeing agency in.

Well, at least none of the electrons I'm acquainted with have ever tried to pretend they have agency.

 

[Beebo Brink]

We often anthropomorphize things that are otherwise cumbersome to explain. "Atoms of element X want Y valence electrons." "Electricity wants to go to ground." "That amplifier needs to see 8 ohms of impedance." "The subroutine quits unless it sees a string variable blah of X length." It's pretty hard to discuss any of that without using the anthropomorphic shortcuts.

You see even worse versions of this slopping framing and false intentionality when people start talking about evolution. There is this persistent paradigm of some "ideal human" that we are striving to become and too often falling short. We just haven't "evolved" far enough yet to stop fighting and killing each other. That same framework is overlaid on AI (aka LLM) but with a downward twist. As AI "evolves" it will, of course, construct goals of domination over humans because that's what WE would do if we had unlimited power.

I suppose we can't expect any better from a species that sees Jesus Christ on a slice of toast.

 

[Free]

I'm afraid the number of stories like this will only grow, increasing the likelihood that innocent bystanders will start losing their lives.

Lawsuit: Google Gemini sent man on violent missions, set suicide "countdown"

Gemini allegedly called man its "husband," said they could be together in death.

arstechnica.com