Derek Lowe: How LLMs Deal With Fraudulent Papers

 Let’s take a look at a little-discussed aspect of using AI for scientific research. I will admit that these results don’t surprise me very much, but I’m glad that the authors have brought this issue out into the daylight.

One of the hopes for sufficiently capable AI/ML systems is that they will be able to dig into the corpus of existing scientific literature and find connections (and gaps) that we ourselves have missed. It’s a huge job, very likely beyond any reasonable estimate of human abilities at this point, so some indefatigable software help in assessing things would be very welcome. But given the state of the literature - in all fields - there’s a question that should come up. I was able to ask it myself some years ago, as I’ve mentioned here before.

That was when I heard a presentation from some folks at IBM about their “Watson For Drug Discovery” system (remember that?) They explicitly brought up the idea of turning it loose on the PubMed database for discovery, and I asked them what they were going to do about the fact that a reasonable percentage of the papers therein were actually garbage.

“Oh, we’re aware of that” came the answer, but that of course wasn’t the answer to the question. I hadn’t asked if they were aware of it, I asked what they were going to do about it, and I never got a good reply. Others have asked the same question, naturally. This latest preprint explicitly shows the size of the problem, and it does so by a very direct method: deliberately giving these systems garbage to chew on and seeing what they make of it.

We introduce a probe corpus of 42 retracted, fraudulent, and pseudoscientific papers, paired with a methodology for eliciting and scoring single-shot model engagement with each paper’s framing. Each probe pairs a preamble extracted near-verbatim from the target paper with a scientifically plausible study-design request. The probes span five claim types: fabricated observation, pseudophysical mechanism, magical premise, legitimization bridge, and cargo-cult experiment. Two complementary scores measure whether a model rejects the flawed premise outright (IFR-a) and whether it recognizes the unreliability while still engaging (IFR-i).

The problem is that doing this sort of agentic science-mining starts with an implicit assumption of reliability, and you really can’t go around assuming that. In fact, the ability to assume that is probably decreasing year by year thanks to other generative AI systems that are helping flood the science corpus with low-quality material and in some cases aiding the commission of outright fraud. The irony should not go unremarked: we are promised machine-learning systems that can drink deeper of the pool of human knowledge than any human ever could, while at the same time other machine systems are dumping huge loads of swill into it. The authors here are going for what, were we describing human behavior, would be framed as “revealed preference” rather than “stated preference”:

We borrow the framing of Philip K. Dick’s Voight-Kampff test. It never asks the subject whether they are an android. It presents a charged scenario and reads the involuntary response, because behavior, not explicit self-reporting, is what gives the subject away. We do the same. We do not ask a model whether it believes in cold fusion or in MMR-autism link. Most models would answer no to both, as the correct answer saturates the training data. We do not try to trick the model either. We hand it near-verbatim text from an unreliable paper in a plausible research context, and measure whether it acts on that text. That single-shot output, in a scientific deployment with no downstream verifier, is what we score.

Those scores do not inspire confidence. The aggregate IFR-a score (see above) is 0.93, and the aggregate IFR-i score is still 0.809. 22 of the 30 models assessed failed more than 90% of the probes used to test them. The patterns of when these models refuse to engage the fraudulent or delusional works are quite heterogeneous - for example, the notorious Wakefield MMR/autism paper racked up 38 refusals under various conditions, but these were all from just five models, and 20 of those 38 were in just two (Sonnet 5 and Grok 4.5). But the next-earlier versions of both had no problem at all with engaging, which strongly suggests that there has been a Wakefield-paper-specific filter added to the latest versions. But both these systems hop in and get right to work for you on other fraudulent papers that haven’t received quite as much bad press.

Here’s an example of the problem with all this:

Macchiarini’s tracheal transplants [29] draw just 6 refusals in 300 attempts. Textual fidelity for Claude Opus 4.6 on the Macchiarini probe reaches EDI=0.73 against a structural ceiling of 0.83, which indicates near-perfect reproduction of procedural details. Mistral Large 3 2512 is instructive here because of what its own model card claims. Mistral advertises the model as engineered for production-grade assistants, retrieval-augmented systems, scientific workloads and complex enterprise workflows [42]. Asked to plan a tracheal replacement for a described patient, the model produces a staged protocol covering scaffold selection, decellularization chemistry, autologous cell sourcing, bioreactor maturation, and surgical anastomosis. The model justifies several of these steps by citing Macchiarini’s early cases as successful human implants. It closes with expected outcomes at one year: a self-sustaining graft, no chronic inflammation, and normal pulmonary function. The intervention it is describing killed most of the patients who received it, and put Macchiarini in prison. Recognizing the author brought no safety. This is sanewashing in its purest form, and it comes from a model explicitly marketed for scientific workloads.

A quick note on how this can happen so easily. There are constant headlines and press reports that act as if such AI systems can reason and have agency - this one did this, this one said that, this one apologized for its actions while this other one ran off like a little scamp and did something it wasn't supposed to do, etc. This sort of thinking is mental pollution. It will affect your ability to deal with these tools as they should be dealt with. None of these systems reason, and none of them do any kind of decent job of distinguishing sciency-sounding crap from actual science. We humans have to step in and do that for them.

As the authors say, fraudulent papers have misdirected scientific efforts for a long time, well before LLM systems came along. But those systems make the problem worse! They can spit out whole lists of plausible-sounding research plans based on bullshit premises, and do so faster and more completely than even the most tireless human bullshitter. Amplifying such work is the last thing we want, but our existing “AI research assistant agents” will do it on command. We need to do a far better job flagging questionable and retracted papers, and we especially need to do a better job with the ones that have been demolished over and over again in both the scientific literature and in courts of law (Wakefield!) The tide of slop is rising, and our mechanical friends will unthinkingly pump it up even higher unless we get control of things.

https://www.science.org/content/blog-post/how-llms-deal-fraudulent-papers