How AI Is Disrupting Medicine

[Beebo Brink]

I really think one of the most important improvements we can make to these AI's is making them able to admit when they don't know something. ...and when to tell the user they are a dumbass.

The fundamental flaw in your statement is that AI don't have intelligence. They can't properly evaluate whether or not they know something, a conclusion that can be reached by basic reasoning.

They are LLMs. They spit out associations between words. There are always words to put together, whether or not they reflect reality.

 

[Beebo Brink]

And now this...
Scientists invented a fake disease. AI told people it was real

[Bixonimania] doesn’t appear in the standard medical literature — because it doesn’t exist. It’s the invention of a team led by Almira Osmanovic Thunström, a medical researcher at the University of Gothenburg, Sweden, who dreamt up the skin condition and then uploaded two fake studies about it to a preprint server in early 2024. Osmanovic Thunström carried out this unusual experiment to test whether large language models (LLMs) would swallow the misinformation and then spit it out as reputable health advice. “I wanted to see if I can create a medical condition that did not exist in the database,” she says.

The problem was that the experiment worked too well. Within weeks of her uploading information about the condition, attributed to a fictional author, major artificial-intelligence systems began repeating the invented condition as if it were real.

 

[GoblinCampFollower]

The fundamental flaw in your statement is that AI don't have intelligence. They can't properly evaluate whether or not they know something, a conclusion that can be reached by basic reasoning.

They are LLMs. They spit out associations between words. There are always words to put together, whether or not they reflect reality.

I know they can't think, but have read that part of why they malfunction so readily is because current training algorithms don't give the model any reward for admitting something just wasn't in their input data. This is something they can at least in principle be made to be better at. ...but that could be wrong, or else a model might be better at it by now.

 

[Argent Stonecutter]

They don't "know" if something is in their corpus or not. There is no mechanism for "knowing" things like that.

The only thing that would be useful, would be to scan the corpus for text similar to the prompt, and if there isn't any, say "I don't think I can help you with that". But that would mean admitting that they were just parody generators based on really inefficient search engines.

 

[Casey Pelous]

I know they can't think, but have read that part of why they malfunction so readily is because current training algorithms don't give the model any reward for admitting something just wasn't in their input data. This is something they can at least in principle be made to be better at. ...but that could be wrong, or else a model might be better at it by now.

The model can't evaluate "Does this make sense?" It has no way of checking for "sense" nor even for "accuracy" --- how would YOU check it if you didn't have a brain? If it finds words that fit, then it "knows" whatever it is because the training data says so. Would you have them Google their answer? We know how well that works! We also know how much more computing power it would take in an enterprise that is already using staggering amounts of processing power to produce, in many cases, vast steaming piles of ... um ... information.

I asked what I thought was a simple question -- what's the best converter for LaTeX to epub3? What followed was four frickin' days of Gemini writing gobs of LaTeX to make my (rather complex) document work with what is, allegedly, the best converter. After four days, it became obvious it was just throwing code at the wall to see if it would stick. I threw in the towel, and damn if Gemini didn't admit that its recommendation for best converter was software that is now about 30 years (!!!) out of date and is incapable of handling the more modern LaTeX commands that make my products kinda snappy, if I do say so myself. Since LaTeX (actually TexStudio, which runs LaTeX) spits out a perfectly good pdf, I finally asked, "Should I be looking at a pdf--> epub3 converter?" "Oh, yes, that's a much better way to go." (In fact, it said it was the Gold Standard way to go; it overuses the term Gold Standard to the point of comedy. It had also assured me that the LaTeX -> epub converter was the Gold Standard, as well as any number of colossally bad solutions it came up with.) A half-hour of trial and error, and I had a beautiful conversion.

If the infernal contraption had one ounce of ability to evaluate, "Does this make sense?" or, "Do I actually know how to do this?", it would have told me, "No, don't convert from LaTeX, convert from pdf." But it doesn't have that capability. It was answering my question the best it could (which was, it turned out, not well at all), but I had asked the wrong question.

 

[GoblinCampFollower]

The model can't evaluate "Does this make sense?" It has no way of checking for "sense" nor even for "accuracy" --- how would YOU check it if you didn't have a brain?

There is a very big disconnect between what I said and your reply. I did not say anything about it checking for "accuracy" or "sense" or anything related. I think you had something you wanted to say and bounced off my comment to say it.

Of course LLM's can't check for "sense" but they can check for existence, which leads to Argent's reply:

The only thing that would be useful, would be to scan the corpus for text similar to the prompt, and if there isn't any, say "I don't think I can help you with that". But that would mean admitting that they were just parody generators based on really inefficient search engines.

Yes, this is very close to what I mean. It needs basic checks for does the data exist. Obviously, that will NOT guarantee if the training data makes sense at all, but I do think it would be a huge improvement if LLM's could just sometimes say "I don't know" because nothing in their corpus was related.

 

[CronoCloud Creeggan]

I asked what I thought was a simple question -- what's the best converter for LaTeX to epub3?

/me raises hand like a teacher's pet. I know...I know!

You're better off taking the PDF and converting THAT. In fact that's what most epublishers do. I think Lulu does too.

Since LaTeX (actually TexStudio, which runs LaTeX) spits out a perfectly good pdf, I finally asked, "Should I be looking at a pdf--> epub3 converter?" "Oh, yes, that's a much better way to go."

Laughs, though I'm surprised TexStudio doesn't have an export to epub option itself. Lyx does. There's also LaTeXML

LaTeXML is a converter that transforms TeX and LaTeX into XML/HTML/ePub/MathML and other formats.

It was answering my question the best it could (which was, it turned out, not well at all), but I had asked the wrong question.

Don't ask AI your LaTeX questions....ask Argent.

 

[Beebo Brink]

Of course LLM's can't check for "sense" but they can check for existence, which leads to Argent's reply:

No, they can't. That's our point. There is NO intelligence, no ability to evaluate whether or not a specific set of data exists or does not exist. There is simply word association: Do these words go together, yes or no?

It's a parlour trick that gives the impression of thought. And apparently it works well enough that people expect some rudimentary thought processes to be part of the process in generating an answer. Until the essentially thoughtless nature creeps in as hallucinations and bogus references and meaningless solutions.

 

[GoblinCampFollower]

No, they can't. That's our point. There is NO intelligence, no ability to evaluate whether or not a specific set of data exists or does not exist. There is simply word association: Do these words go together, yes or no?

It's a parlour trick that gives the impression of thought. And apparently it works well enough that people expect some rudimentary thought processes to be part of the process in generating an answer. Until the essentially thoughtless nature creeps in as hallucinations and bogus references and meaningless solutions.

I think you're partly correct in that yes, LLM's have no intelligence. LLM's are inclined to just hallucinate when asked a nonsense question. That IS true.

The part I think we are disconnecting on is that most chat bots are NOT just naked LLM's. They are full of cases where the human programmers built in special cases to stop the LLM's from doing their stupid thing. I think that what I'm proposing is already in progress. For example, I just asked this to chatGPT to test it:


What did Shakespeare have to say about Krampus?

"He didn’t — Shakespeare never mentions Krampus. Krampus is an Alpine folk figure (a horned, punitive companion to Saint Nicholas) from Central European tradition; the specific Krampus motif became prominent later than Shakespeare’s lifetime. Shakespeare’s plays do include devils, demons, and mischievous Christmas/folk figures (e.g., Puck/Robin Goodfellow), but there’s no direct reference to Krampus in his works."



In this example, chatGPT didn't just hallucinate nonsense, it correctly told me that my question didn't make sense. I suspect, but can't prove, this is because the programmers of chatGPT are at least trying to do what I'm talking about. I know their success is mixed so far and you can still trick these bots in a million ways, but I do think it is possible for the programmers to build in more and better checks to prevent blatantly nonsensical slop.

What I'm talking about is pretty analogous to the plethora of content filters already in place that stop most of the bots from writing smut for people. ....now..... obviously there are ways to trick these filters, but I think the basic filters are possible.

 

[Free]

LLM's are inclined to just hallucinate when asked a nonsense question. That IS true.

They also hallucinate when asked rational, direct questions. "How many r's are in the word strawberry?" is not nonsensical. It's a pretty normal. And until pretty recently (I just checked), ChatGPT would consistently get the number wrong. Programmers had to build in an evaluation of sorts to stop it from "doing their stupid thing" because it wasn't attempting to evaluate how many times the letter r existed in the word strawberry in the way we would do it, or even in the way a typical piece of software would. And that is the ongoing issue with LLMs.

 

[GoblinCampFollower]

They also hallucinate when asked rational, direct questions. "How many r's are in the word strawberry?" is not nonsensical. It's a pretty normal. And until pretty recently (I just checked), ChatGPT would consistently get the number wrong. Programmers had to build in an evaluation of sorts to stop it from "doing their stupid thing" because it wasn't attempting to evaluate how many uses of the letter r existed in the word strawberry in the way we would do it, or even in the way a typical piece of software would. And that is the ongoing issue with LLMs.

I agree with you. I wasn't defending their overall use. I avoid them as much as possible in most cases. All I was trying to day is that it IS possible to make one of them check if data exists. ...and I said I would expect it to get it wrong sometimes.

 

[Casey Pelous]

There is a very big disconnect between what I said and your reply. I did not say anything about it checking for "accuracy" or "sense" or anything related. I think you had something you wanted to say and bounced off my comment to say it.

First time that ever happened, honest!

I think I skipped a step -- You were saying, to paraphrase, LLM's should say, "I can't answer that" if they don't know the answer to something. In my mind, that would be the machine solving the problem, "Does this answer I have just concocted 'make sense.' Do I actually possess the accurate information I need to correctly answer this question? Does this answer align with the highest quality information on this topic?" That was how I understood your post. Apologies for missing the mark.

My experience with -- and very limited understanding of -- AI really suggests that something is fundamentally flawed in the underlying model that makes it fiendishly difficult to solve that problem dependably. It certainly was the case in the event I talked about. Ideally, I suppose Gemini would have set up a LaTeX compiler -> epub converter -> epub evaluator and just run iterations until it found a combination that worked, but that's just a dumbass brute-force attack: inelegant, inefficient, and utterly impossible at this time.

I rather suspect that if you could work out an algorithm to solve what I'll call the "Information Quality" problem, you would become both wealthy and legendary.

 

[GoblinCampFollower]

I rather suspect that if you could work out an algorithm to solve what I'll call the "Information Quality" problem, you would become both wealthy and legendary.

Thanks for the reply, and yeah... I know it's a hard problem and a very broadly defined one. I don't think the strong form of it "does this make sense?" Is at all solvable. I think we all agree on that point. My own hypothesis, which is much more modest, is just "Does my training data contain a direct reference to what you asked?" I think that is already partly solved, and will get incrementally better. ....but is still a difficult problem for sure.

 

[Argent Stonecutter]

The part I think we are disconnecting on is that most chat bots are NOT just naked LLM's. They are full of cases where the human programmers built in special cases to stop the LLM's from doing their stupid thing.

But those things don't actually make them more useful than a search engine because those are the same kinds of things search engines do to combat SEO. The supposedly useful part of the chat bot is the LLM part. And that's rubbish.

 

[Beebo Brink]

The part I think we are disconnecting on is that most chat bots are NOT just naked LLM's. They are full of cases where the human programmers built in special cases to stop the LLM's from doing their stupid thing.

Didn't you just prove my point? Building in those special cases is not increasing the reliability of AI; it's just adding another bucket to bail water. When AI goes off the rails, the developers add yet another rule to correct for that specific situation. The process itself isn't getting better.

 

[GoblinCampFollower]

Didn't you just prove my point? Building in those special cases is not increasing the reliability of AI; it's just adding another bucket to bail water. When AI goes off the rails, the developers add yet another rule to correct for that specific situation. The process itself isn't getting better.

I agree with you. You were misunderstanding my point. I was NOT trying to argue this is the same thing as the AI itself learning or getting better. You refuted a point I wasn't trying to make.

EDIT: I looked back through some of my posts. I can see why my wording was unclear. I always thought of this as something the human programmers were making the LLM do. I can see why you thought I meant the LLM would learn to do this, which was obviously not going to happen. I've said before on other threads that I think the LLM's themselves have plateaued a long time ago. The new "innovation" involves a lot of human programmers shoving in special cases.

EDIT2: fixed a spelling error.

 

[Free]

miss understanding my point.

Ahh, Miss Understanding. I've found her to be a lovely person. But I don't really know her that well.

 

[GoblinCampFollower]

Ahh, Miss Understanding. I've found her to be a lovely person. But I don't really know her that well.

LOL... yeah... hasty early morning typing with no coffee is no good...

 

[Free]

If it's on the internet, it's got to be real. Isn't that right, A.I.?

Scientists invented a fake disease. AI told people it was real

Got sore, itchy eyes? You’re probably one of the millions of people who spend too much time staring at screens, being bombarded with blue light. Rub your eyes too much and your eyelids might turn a slight, pinkish hue.

So far, so normal. But if, in the past 18 months, you typed those symptoms into a range of popular chatbots and asked what was wrong with you, you might have got an odd answer: bixonimania.

The condition doesn’t appear in the standard medical literature — because it doesn’t exist. It’s the invention of a team led by Almira Osmanovic Thunström, a medical researcher at the University of Gothenburg, Sweden, who dreamt up the skin condition and then uploaded two fake studies about it to a preprint server in early 2024. Osmanovic Thunström carried out this unusual experiment to test whether large language models (LLMs) would swallow the misinformation and then spit it out as reputable health advice. “I wanted to see if I can create a medical condition that did not exist in the database,” she says.

The problem was that the experiment worked too well. Within weeks of her uploading information about the condition, attributed to a fictional author, major artificial-intelligence systems began repeating the invented condition as if it were real.

The following is the saddest part.

Even more troublingly, other researchers say, the fake papers were then cited in peer-reviewed literature. Osmanovic Thunström says this suggests that some researchers are relying on AI-generated references without reading the underlying papers.

 

[Cristiano]

They also hallucinate when asked rational, direct questions. "How many r's are in the word strawberry?" is not nonsensical. It's a pretty normal. And until pretty recently (I just checked), ChatGPT would consistently get the number wrong. Programmers had to build in an evaluation of sorts to stop it from "doing their stupid thing" because it wasn't attempting to evaluate how many times the letter r existed in the word strawberry in the way we would do it, or even in the way a typical piece of software would. And that is the ongoing issue with LLMs.

Wasn't it the word raspberry?