Detecting health and science misinformation with AI

Misinformation is everywhere, and spotting it can be difficult. But with the advent of AI, it could soon be easier to separate fact from fiction.

IBM Research scientist Yufang Hou and her collaborators are working on an AI tool called MISSCI for automatically detecting when scientific information is misused to make false claims. They’re presenting their work this week at the Association for Computer Linguistics (ACL)’s annual conference, taking place August 11 to 16 in Bangkok, Thailand.

Health and science misinformation has been on the rise since before the COVID-19 pandemic, but in many ways it’s more sophisticated than ever. It doesn’t just come from social media or dodgy websites with odd domain names, either — it can find its way into the pages of mainstream media outlets thanks to savvy actors who cite real academic papers to burnish their credibility. It’s disturbingly easy for these convincing-looking claims to fool reporters and editors, who inadvertently signal-boost false information.

Hou and her collaborators from the Technische Universität Darmstadt and Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) saw this problem as a great opportunity to bring modern AI tools to bear. For most people, it’s simply unrealistic to investigate the factual basis of every health claim we read. Chasing down all the footnotes and references in an article can be overwhelming, even for a doctor or scientist who knows how to parse that kind of information. AI, though, is perfectly capable of sifting through loads of data and identifying patterns.

Enter MISSCI, a new argumentation theoretical model designed to tell whether scientific evidence actually supports a claim. It’s been applied to annotate a corpus of 8,695 references from 527 articles on the fact-checking website HealthFeedback is a website that reviews the accuracy of health claims from social media and news reports. It's part of the editorially independent non-profit Science Feedback, whose funders and partners include Meta, TikTok, the Poynter Institute, the University of California Merced, the European Media and Information Fund, the Google News Initiative, and others.HealthFeedback, focusing only on those ones that addressed inaccurate medical claims. For a suspicious claim that cites a scientific study, the model retrieves any relevant evidence from the study, and it determines whether the evidence supports the claim or contradicts it. If the evidence contradicts the claim, the model explains why the claim is false by detailing the fallacious reasoning process. The researchers have developed a computational model to solve these tasks by using a large language model (LLM). The computational model is still in the prototype stage and can’t yet do this fully on its own with high precision, Hou says. But parts of it may already make a time-saving research tool for human fact-checkers.

Among the HealthFeedback articles, the team found 208 links that misrepresented the scientific papers they pointed to.

This research began in 2022, with the team’s early efforts to use natural language processing technologies to detect health misinformation. Despite public interest in automated fact checking, though, the team found one major obstacle: The counter-evidence was not available. To verify whether a statement is factual, it’s necessary to have the facts, but since false claims are usually detached from actual evidence, finding the true version can be an impossible task.

“Sometimes there’s no off-the-shelf counter-evidence fact-checkers can point to,” Hou says. Here’s how that fact-checking process normally works:

When a fact checker investigates a scientific claim, they dig into the evidence that’s referenced. One example the team used in the paper was a news article claiming the drug hydroxychloroquine can treat COVID-19 infections. The article cited a real scientific paper, which showed hydroxychloroquine can treat the virus SARS-CoV-1 in a dish of cultured cells. This study appears to support the claim, but there are two crucial issues. The first is called a fallacy of composition, the logical fallacy that a part of something (cells, in this case) will behave the same way as something made up of cells (the human body). Scientists do not consider an experiment involving cultured cells to be solid evidence that a treatment will work in people, even if the cells are from people. This is why part of the process of a drug trial involves testing a medication in many people before it can be called safe and effective.

The second issue is subtle but equally misleading: false equivalence. SARS-CoV-1 isn’t the virus that causes COVID-19 — that would be SARS-CoV-2. To the casual reader, this might seem like a minor detail, but to scientists, it’s the difference between fact and fiction.

The existing automated fact-checking models struggle to detect these types of fallacies, Hou says, because there may be no reliable source of data that directly states the opposite — in this case, that hydroxychloroquine is not a cure for COVID-19. And because misinformation is meant to look credible, there’s nothing on the surface that an automated classifier would flag as fallacy: In that example paper, a sentence actually states that hydroxychloroquine treats coronavirus infection. Given that SARS-CoV-1 is a coronavirus just like SARS-CoV-2, to a classifier including the one based on LLMs the statement would be technically true, but it isn’t evidence that the drug can treat COVID-19.

The research team wanted to design MISSCI and train the corresponding computational models to follow the routines of human fact checkers and actually trace a false claim back to the evidence, creating a road map that shows how valid information is twisted into misinformation.

Using a body of articles from HealthFeedback, they built multiple models to reconstruct the fallacious reasoning process of misrepresented claims and explain why they are untrue. They tested a Llama 2-70B model and GPT-4, two different LLMs. In their experiments, GPT-4 performed better at these tasks but still fell behind humans’ performance.

“If we provide a model with the false premises written by professional fact-checkers, though, both models perform decently,” Hou says. For example, GPT-4 performed with about 77% accuracy. Under the more realistic scenario where the models are asked to reconstruct the whole picture, the best performance was about 31.7% from GPT-4. When the researchers asked the models to produce false scientific statements, GPT-4 performed well according to human evaluators, but Llama 2 basically couldn’t do it.

Interestingly, if they asked the models to first give a reason that a statement was false and then give a prediction of the fallacy type, they performed better. “If you force a model to think about what’s wrong, it’ll give you better performance with the final classification results,” Hou says.

Ultimately, this new work suggests that even though machine learning may eventually be able to help automatically detect scientific misinformation, it isn’t yet up to the task. Other scientists at IBM Research have previously found how easy it can be to trick machine vision models with fake stock market information, emphasizing just how crafty people can be when it comes to tricking each other and machines.

Hou and her collaborators are following up on their work in another paper that’s currently under review, by removing constraints of using the paraphrased version of the publication context and adding the original paper as inputs to the model. They first feed in a questionable claim as an input, and the model searches the whole scientific paper that the statement is supposedly based on. It then analyzes to what extent the information is likely based on some piece of information from the paper. Then passages from the paper are used as additional inputs for determining factuality.

“I do think some of our solutions could be readily deployed in reality,” Hou says. On the whole, MISSCI isn’t ready to be rolled out as a standalone tool for fact checking, but in practice, parts of it could be used as AI tools to aid research. Specifically, the part of the work that retrieves relevant information from the cited paper could help a human fact checker chase down the facts with less effort.